Faculty of Informatics and Information Technologies -...

Slovak University of Technology in Bratislava

Faculty of Informatics and Information Technologies

FIIT-5212-5734

Ján Podmajerský

Interactive Whiteboard

Bachelor thesis

Degree Course: Informatics

Field of study: 9.2.1 Informatics

Place of development: Institute of Applied Informatics, FIIT STU Bratislava

Supervisor: Ing. Andrej Fogelton

2013, May

ANOTÁCIA

Slovenská technická univerzita v Bratislave

FAKULTA INFORMATIKY A INFORMACNÝCH TECHNOLÓGIÍ

Študijný odbor: Informatika

Autor: Ján Podmajerský

Bakalárska práca: Interaktívna tabul’a

Vedúci bakalárskej práce: Ing. Andrej Fogelton

máj, 2013

Bakalárska práca sa zaoberá vytvorením dostupnej verzie interaktívnej tabule. Najzaují-

mavejšie je použitie detekcie LED diódy vo viditel’nom spektre svetla, pretože väcšina dos-

tupných riešení používa infracervené spektrum na dosiahnutie požadovaných vlastností. Táto

LED dióda je umiestnená v pere. Používa sa na interakciu s vysielaným obrazom z projek-

tora. Pre využitie viditel’ného spektra sme sa rozhodli, najmä kvôli možnosti používania

tohto riešenia aj za denného svetla, v bežných podmienkach. Toto svetlo je snímané bežne

dostupnou webovou kamerou. Na získanie polohy LED diódy sa používa prahovanie kanálov

HSV modela. Po detekcii svetla sa aplikuje homografia, ktorá umožní prevod medzi súrad-

nicovými sústavami kamery a zobrazovacieho zariadenia. Následne sa vd’aka operacnému

systéme umiestni kurzor myši na požadované miesto. Dosiahli sme presnost’ približne 5

pixelov. Naše riešenie nefunguje v extrémnych svetelných podmienkach, akým je priame

žiarenie slnka na projektor. Vytvorili sme prototyp pomocou OpenCV a Qt.

ANNOTATION

Slovak University of Technology Bratislava

FACULTY OF INFORMATICS AND INFORMATION TECHNOLOGIES

Degree Course: INFORMATICS

Author: Ján Podmajerský

Bachelor Thesis: Interactive Whiteboard

Supervisor: Ing. Andrej Fogelton

2013, May

This bachelor thesis deals with construction of an inexpensive interactive whiteboard. The

most distinctive feature of the solution is a light emitting diode(LED) detection, which is

done in the visible light spectrum. Most of available solutions use infrared light to achieve

the expected behavior. Having taken into consideration the state of the art, we have decided

to use the visible spectrum of light, because it can it can work in common daylight conditions

too. The LED is mounted at the top of the pen, which points onto a projected image of the

video projector. Infrared light cannot be used on a daylight, because infrared light is present

in daylight, due to which the use of infrared light could be inaccurate. With the aim of

inexpensive solution a common webcam can be used as the capture device. Thresholds in

individual channels of the HSV color model are set up to filter the LED. The evaluation can

provide a real-time performance. Homography is used to translate the camera coordination

system into screen coordinates. Finally the operating system’s interface places mouse cursor

on demand position. We achieved precision within 5 pixels. The detection does not work in

the extreme light conditions, when there is a strong sun shining on the projecting plane. We

created a prototype implemented in OpenCV and Qt.

Declaration of Honor

I honestly declare, that I wrote this thesis independently under professional supervision of

Ing. Andrej Fogelton with citated bibliography.

May, 2013 in Bratislava signature

Acknowledgement

My thanks mostly belong to my supervisor Ing. Andrej Fogelton for his willingness, profes-

sional support and helpful advices during the work in this thesis. I thank also to Ing. Michal

Kottman for his advice about using new technologies in the user interface. I cannot forget

about my family and friends who supported me during the work in this thesis.

Contents

1 Introduction 1

1.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 State of The Art 3

2.1 Wii Remote, Infrared Camera . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.2 Wii Remote Based Complete Solution . . . . . . . . . . . . . . . . . . . . 4

2.3 Wii For Tracking Objects In 3D Space . . . . . . . . . . . . . . . . . . . . 4

2.4 New Trends in Interactive Whiteboard Development . . . . . . . . . . . . . 5

2.5 Fingertip And Eye Tracking . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.6 Using Multi-camera System . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.7 Interactive Whiteboard Using Common Camera and LED Light . . . . . . 8

2.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 Analysis of Used Principles 11

3.1 OpenCV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 Qt Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.3 HSV Color Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.4 Homography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

4 Interactive Whiteboard Prototype 15

4.1 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.2 Camera Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.3 Automatic Calibration Process Using Chessboard . . . . . . . . . . . . . . 17

4.4 LED Pen Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.4.1 LED Pointer Construction . . . . . . . . . . . . . . . . . . . . . . 20

4.4.2 Color Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Contents Ján Podmajerský

4.5 Cursor Position Localization . . . . . . . . . . . . . . . . . . . . . . . . . 22

5 Testing 25

6 Conclusion 27

A Technical Documentation 31

B User Guide 35

C Resumé 37

D DVD Contents 41

ii

Chapter 1

Introduction

These days there are many presentations, we live in an information society, every lecture atthe university consist of presentations. Most of the speakers need to emphasize some of thepresenting ideas so they usually use accessories to help them display important notes.

Blackboard has been used in presentations for a long time. It allows a speaker to note thekey elements of the presentation, also the viewers can put some notes down while seeing it.This is suitable only for short and not complex presentations, but when it comes to biggerand more complex presentations, this tool is not the ideal for providing a pleasant help. Thebasic ability provided by blackboard is to write notes with chalk to emphasize the importantpoints from the lecture. However chalk produces dust, which may be dangerous for peoplewith allergy, also it is not comfortable to get dirty of chalk dust. Later whiteboard wasintroduced, but it has similar abilities as blackboard, except the dust problems. Howeverother problems occur, for example markers cannot get dry, it requires regular use, moreoverit is more expensive to use markers.

The apparent improvements in presenting were made with digital video projectors introduc-tion. This tool can easily be connected to the speaker’s computer, which allows him to createa more interesting presentation. However the interaction with the presentation is possibleonly via notebook. If the speaker wants to edit the presentation, he or she has to use com-puter’s keyboard or mouse. Moreover the interaction with the audience is a bit complicated,because using the computer of someone else might be difficult, having different settings. Itis easier to interact directly with the screen where the presentation is on.

It is appreciated to have an interface, which allows the speaker to interact with the computerand the whole presentation more naturally, to write and paint more easily. The interactivewhiteboard is a system providing all the desired features. It allows speaker to write directlyon a projected image, simply having a big touchscreen. Touchscreen is the most natural in-terface, one just touch the screen and manipulate directly with displayed objects, the systemregisters the interaction and apply changes in real-time, so it creates real-world system infront of the user. It is natural to touch the things which people want to work with. As aresult, using such a system is easy and intuitive for everyone. However all the interactivewhiteboards do not use a huge touchscreen, but they consist of separate devices for project-ing images and registering moves. The interactive whiteboard is a system usually consistingof a digital video projector, special whiteboard and a special pen, communicating with thesoftware. There are different configurations and versions, even different technology may beused, in some cases, no special pen is required and the presenter can use own fingers. This

Chapter 1. Introduction Ján Podmajerský

behavior is achieved by a special material from which whiteboard is made, for instance theconducting backplate. When the user touches the screen in this configuration, the desiredposition is electronically set and registered by the driver, which can create simple mouseevent or even a multi-touch. Other configurations of interactive whiteboard are equippedwith electromagnetic wires behind the screen, which interact with passive pen, having a coilin the tip, for example. Thanks to the magnets, it is easy to determine the position of the penand send the information to the driver. Unfortunately all the configurations are expensive, upto 1700 e. This disadvantage prevents from massive use of schools and other institutions.Moreover it is very robust and static, so interactive presentations can be done only in roomsequipped with interactive whiteboard.

Despite commercial interactive whiteboard providing good interaction, it is not worth to buy.The main reason is all the necessary components for construction are available separately.Image processing and computer vision techniques can be used to provide such a solutionbased on those common devices, but it would be much cheaper. Digital video projector canbe used for image projection on any plane and a common webcam can capture light pen infront of the plane, creating a touch screen. The only unavailable component is a software,which could transform signal from the devices to control the communication between them.However the software could be implemented using computer vision techniques, finally everywhiteboard (wall) could be transformed into an interactive one. The speaker would need justa light pen and the software, other crucial components are easily available. So the robustcommercial solution would be reduced just to few off-the-shelf accessories, which are easyto transport, such an interactive whiteboard would be portable, not restricted to use in specialrooms.

Few pioneers already created inexpensive solutions of an interactive whiteboard. Howevermost of them uses infrared camera, which brings additional problems of interference due tothe infrared light obtained from the sun, so their solution can be only used in rooms with lowsunlight. Moreover advanced infrared cameras are not so commonly available.

1.1 Requirements

We plan to use interactive accessories for presentations. The goal of this solution is to createa software, which can track light emitting diode (LED) pen by off-the-shelf webcam. Thisposition would finally be translated to the coordinate system of a computer and move mouseto the place where LED pen was detected. We want our interactive whiteboard solution to beavailable to everyone. Consequently our solution would consist of off-the-shelf components:

• webcam,

• digital projector,

• pen with LED at the tip,

• application on LED detection with real-time performance.

2

Chapter 2

State of The Art

There are only a limited number of low-cost interactive whiteboard solutions. Despite theinfrared light having some disadvantages it is still used in most of these cases for detection.Implementation of a whiteboard with infrared light detection reduces its features, becausedaylight consists of an infrared light, so these whiteboards can be used only in rooms withlow sunlight. Daylight can interfere infrared light pointer detection. We analyze availablelow-cost solutions of interactive whiteboard.

2.1 Wii Remote, Infrared Camera

The severe development in the area of low-cost interactive whiteboard was made by JohnnyChung Lee, who released a software [Lee et al., 2007], which demonstrates usage of infraredlight for tracking. Using this principle, interactive whiteboard can be created intuitively, froma digital video projector and Wii Remote. Since the Wii Remote tracks sources of infraredlight (detects infrared LED) the software know the exact position of the LED pen, conse-quently there is no problem to set mouse on demand position within an operating system.Unfortunately this solution is not widespread because of higher hardware expenses (Wii isnot commonly available).

Wii Remote was released in 2005 by Nitendo and became easily popular allowing usersto interact naturally with the Nitendo console by their own body. It is a motion detectiondevice having a high quality infrared camera, capturing at 1024 × 768 resolution with built-in hardware blob tracking of up to 4 points at 100 Hz. The device is connected to Nitendoconsole via Bluetooth on 2.4 GHz band. Wii Remote can communicate the same way alsowith the computer. These features led many developers to use Wii Remote for motion andlight detection.

However there is a considerable issue in detecting conditions, infrared light is part of day-light, so using this camera in daylight, in normal conditions would confuse the detectionalgorithms and infrared pointer would not be detected properly. This problem can easily besolved while using the Wii Remote together with Nitendo Console and Sensor Bar. SensorBar contains fluorescent lights, which emit little infrared light, so there is no interferencebut high positive detection rate. However there is no possible extra light emission near theinfrared pointer. It can only be used in rooms, where there is no to little sunlight present.

Chapter 2. State of The Art Ján Podmajerský

2.2 Wii Remote Based Complete Solution

Johnny Chung Lee created a software [Lee, 2008], which influenced many other developersin human computer interaction. His software detects infrared LED installed on a pen, usedas a pointer in speaker’s hand. He also created a tutorial showing how to construct such a penat home, just for a few euros and little electronics skills. It is intuitive and easy to interactwith, the behavior is the same as if the whiteboard was touchable. The core of the solutionis the Wii remote, which captures the interactive area. However there are few problemswith shadowing, so the Wii Remote is usually placed right onto the projector, minimizingthe problem. The speaker wants to see where the pointer points on a whiteboard, so he orshe stands in front of the projector carefully, giving attention to the projector, so no extraattention is required to avoid shadowing of the camera (Figure 2.1).

Figure 2.1: Johnny Lee’s Wii Remote solution1.

As Wii Remote can track up to 4 points, there is no problem to create a multitouch white-board. The speaker can use 2 infrared pens and the software creates accomplished multitouchscreen in this solution. Moreover one does not have to use a projector at all, the user can sim-ply use computer monitor instead, so the multitouch screen is created.

2.3 Wii For Tracking Objects In 3D Space

Free hand interface for controlling applications based on Wii Remote infrared sensor [Linet al., 2010]. The scientists from Japan developed accomplished solution for interaction withcomputer just with hand waving. There is a possibility to create user’s own gestures formanipulating with the computer. This solution uses Wii Remote and its infrared camera todetect user’s motion via the reflection of the infrared light emitted on a user’s hand. Thesoftware transforms the rays, detects the gesture and sets the computer according to thecreated gesture. It is still in development, some problems may occur and the interaction withmouse and keyboard is required.

The solution requires a computer, Wii Remote and LED panel consisting of hundred LEDsemitting infrared light in the user’s hand. This panel has to be placed together with Wii

1http://wiiwhiteboard.wordpress.com/

4

http://wiiwhiteboard.wordpress.com/


Remote in front of the user (Figure 2.2). The panel is rectangular-shaped with a hole in themiddle, which is just as big as the Wii Remote could be placed in it. The reflection of infraredrays makes the best position for the panel above, or below the computer monitor to use thissolution naturally. Rays of infrared light from LED panel emission impact on the user’shand, few are reflected back and captured by Wii Remote, consequently they are detectedand attached to a gesture by the software. Interaction with this system is intuitive, there is asoftware application, registering gestures in advance, which could be used for interaction.

Figure 2.2: Wii Remote detecting hand [Lin et al., 2010].

All the advanced processes made in the software are possible because of Managed Libraryfor Nintendo’s Wiimote2, the final output is an easy-to-use managed API for the Wii Remote.

This solution shows usage of light reflection, but it means an extra panel to use while creatingan interactive whiteboard.

2.4 New Trends in Interactive Whiteboard Development

There are many developers, but most of them use special cameras, or the usage of the solu-tion is restricted to special light conditions, which are not easy to achieve. However a fewsolutions require no special camera. Sometimes even no extra light pointer, for instance ahigh contrast marker pen (red, green...) is sufficient. The only necessary components arecommon webcam, digital video projector and a pointer. When it comes to functionality, thedetection algorithms could be mistaken, because the color of a pointer can be displayed onthe screen, but more elaborate algorithms can solve this problem.

Mando design3 is one of the elaborate solutions. It uses a contrast marker pointer. The systemconnects a projector and a camera which is placed in front of the whiteboard (Figure 2.3). Aspeaker stands naturally in front of the whiteboard holding marker, the software detects theobject and moves the mouse.

The most important task is to create a model of a pointer to track. At first the pointer isscanned and registered in the system, its colors are hashed. During the detection the hashedcolors of registered pointer are compared to the captured image. Thresholding method is used

2http://wiimotelib.codeplex.com/3http://mando.sourceforge.net/

5

http://wiimotelib.codeplex.com/

http://mando.sourceforge.net/


for detection the pointer. If all of those pixels are found in the captured image, it is detectedas the pointer. There are still some problems to simulate mouse events, for instance themouse left click is simulated by stable marker position. It is not working properly, becausemany times a speaker wants just to point somewhere with the pointer, so he holds it stableand the solution clicks.

Figure 2.3: Detection of a marker4.

2.5 Fingertip And Eye Tracking

In development of interactive whiteboard, tracking different pointers can lead to create newforms of communication and interaction with computer. Using no pointer and tracking faceas well as hands are deeply described in [Cheng and Takatsuka, 2005].

The solution uses most of the components, as previous ones, but uses them in different andmore interesting way. A whiteboard is placed in front of a video projector, or large-screendisplay can be used, too. A speaker stands in front of the projected screen, standing a fewmeters back from it. The speaker uses no pointer, he points just with own fingers. Fingersare tracked by a camera, which is placed above the whiteboard, scanning not only the user’shand but also his face. The user feels like interacting with a transparent touch screen, andthe screen can be a meter behind him (Figure 2.4). The camera is common, off-the-shelf.Thanks to the availability of the components there is a potential spreading in the future.

To determine where the user is pointing, it is important to construct the line in 3D space,which is possible by connecting at least two points within the space. In this case one pointis user’s fingertip and the other is the user’s eye, at the end of the line there is the demandedcursor position on the screen. However finding those points can bring few problems, becausethis solution uses just one camera and no pointer. The solution has to determine where theuser is looking and pointing with his or her finger. This detection is done by elaboratetechnique. The first part is to detect the position of the eyes and fingertip by computer visionalgorithms. Those coordinates are only defined in two dimensions, but it is important toknow also the third coordinate of those points in order to create line in 3 dimensional space.The only possible solution is to approximate the coordinates, but it can be counted reasonablyprecise thanks to general rules of object detection. Objects closer to the camera are bigger,those further are logically smaller Assuming all the users have the same face width, which is

4https://vision.eng.shu.ac.uk/mmvl/viewfinder/report_roldan.pdf

6

https://vision.eng.shu.ac.uk/mmvl/viewfinder/report_roldan.pdf


based on a survey of people’s face width, the width is set to 17 centimeters. It is also possibleto measure the user’s face in the initialization procedure. Therefor the distance to the cameracan be calculated.

After applying previous rules, all the coordinates can be created. Face detection has stillsome restrictions, but in common conditions this solution works. The solution uses OpenCVlibrary, because real-time fast measurements are required.

Figure 2.4: Fingertip and eye tracking [Cheng and Takatsuka, 2005].

There are still limitations with the position of the camera, because the user looks naturally tothe screen not to the camera above, so the position and angles of the camera capturing haveto be decided carefully. If they are not estimated correctly, the whole solution will not work.However it shows interesting aspects of detection and completely different ideas in creatinginteractive whiteboard. It is more suitable for small class presentations, not big promotion.

2.6 Using Multi-camera System

The solution shows how to construct a multitouch-display technology using multiple cam-eras [Korkalo and Honkamaa, 2010]. They are placed at the top of the screen. Their opticalaxis is parallel to the screen and are placed as close to touch-surface as possible, so they haveto use wide angle lenses (Figure 2.5). More cameras are used to get higher accuracy, theyare not synchronized, so images are captured at higher rates. Calibration of the cameras isdone manually. Fingers are detected thanks to active light bars made of polished Acryl rods,which are placed on the sides and bottom sides of the screen. The bars have LEDs at eachend and also a narrow tape on the sides, which creates detectable light to the cameras. Thelight from bars creates a light field, which can be broken when fingers enter the screen tointeract. Consequently the position of the fingers can be determined thanks to image pro-cessing. Every image from the camera is thresholded in order to find light bar. But there aresome spaces in the bar, which are finger’s position.

7


Figure 2.5: Mutliple cameras detecting screen [Korkalo and Honkamaa, 2010].

2.7 Interactive Whiteboard Using Common Camera andLED Light

Design of an interactive whiteboard, using inexpensive off-the-shelf cameras, neither ex-tended nor professional [Jantz et al., 2007], which provides flexible interactive whiteboardalternative.

This solution is based on an idea of constructing mobile and flexible interactive whiteboard,the authors also concerned about the price of the final product. They use common com-ponents off-the-shelf webcam and digital video projector. The key element is to create apointing device, now it can be any light emitting device, for instance a laser pointer or akeyring pendant with an LED. There is no problem to obtain such a pointer.

Projecting image is captured by the camera, placed next to the projector, to avoid shadowingby the speaker (Figure 2.6). Apparently there are two different coordinate systems, oneis camera system other the screen system, different angles between projector and cameracause this problem, the communication between these 2 systems is crucial to create real-time detection application, to place the mouse cursor on the right place. This problem can besolved by calculating the screen coordinates and camera coordinates. It is done by calibrationmechanism, which displays grid of 3× 3 dimensions and force user to point by the LED penat each of the crosses placed in the grid. It is not convenient to force the user to point atthe screen while he could speak about his presentation. Moreover it is possible to calibratethe system automatically by projecting high contrast colors. The necessary camera colorsetting, amplified colors and reduced brightness or saturation, enables real-time interactivewhiteboard construction. However it also requires image processing algorithms to prepareeach image from the camera for light tracking. When the light is filtered, there is no singlepoint detected, but more noise around. The mouse is set at the centroid of those points. Whensetting every single image it uses fast data structures to minimize computer processor usageto allow to run other applications smoothly.

This solution has the quality to be used in any presentation. It is flexible, easy and intuitiveto use. The speaker needs just to connect all the components, which are used also duringcommon, non-interactive presentation. Put them in the right place and take an LED pointerand the presentation can start.

8


Figure 2.6: Our future solution [Jantz et al., 2007].

While presenting the speaker stands in front of the screen holding only LED pointing de-vice, which is used for natural pointing at the screen, it is captured and translated into thecoordinates of the pointer to the computer. Finally mouse cursor is placed there.

2.8 Conclusion

In this thesis we provide inexpensive solution for presenters and teachers, which could beused as a software for interactive whiteboard, while common light conditions are met. Com-monly used infrared light would be replaced by visible light. The solution would consist ofan off-the-shelf webcam, light emitting diode (LED) pen and a digital video projector. Anyscreen could be transformed into a touch screen with this software and the LED pen. Wedecided for implementation in OpenCV, because of high performance and wide variety ofalgorithm support. The best position of the camera depends on the primary hand, used by aspeaker. To avoid shadowing the pointer and to capture images precisely without deforma-tion, the angles between webcam and the projected image are important for the quality of thedetection. The speaker should have a pointing LED pen in hand using it the same way as ina commercial interactive whiteboard solution. While the speaker should point to the slidesby the LED pen a webcam should capture the screen and the software should detect the lightand counts the coordinates and moves a mouse on a computer to the desired position. All ofthese actions should be projected on the screen real-time, no latency is expected. We designa solution which would enable interaction with projected images, similar to the commercialinteractive whiteboard with reduced functionality. We plan to experiment with different lightconditions to reach a high detection rate and consequently a convenient interaction.

The main advantage of this solution would be its availability, by using off-the-shelf hardware.Connecting to a video projector is not a problem in a presenting room. The only thingusers would need is LED pen, which is inexpensive and construction does not require highelectrotechnics abilities.

9


10

Chapter 3

Analysis of Used Principles

In order to construct interactive whiteboard, it is important to produce real-time software forLED detection. After analyzing the-state-of-the-art of software interactive whiteboards wedecided to use common methods for detection and user interface, too. In this chapter weintroduce the technologies used within the prototype.

3.1 OpenCV

OpenCV is an open source computer vision library. At first it was an initiative of Intel [Brad-ski, 2008], later several universities cooperated in development. However the usage of thelibrary was restricted to just a small number of people, who conducted research into computervision domain in the specific universities. It allowed them to create high-quality applications,and finally more and more people cooperated on the library to do public release. At first itwas implemented in C, since the version 2.0 C++ is the main language. However OpenCVcan be also used in different languages, for instance Phyton, Ruby, etc. We implemented oursoftware in 2.4.3 version, which completely covered object oriented paradigm C++.

OpenCV was released in January 1999, since then it is under continuous development. Itprovides many implemented image processing algorithms. However the OpenCV library issuitable also for different tasks, for music recognition, where vision recognition techniquesare applied to sound spectrogram images, for example.

Despite being open source library it can be used in commercial products without any obli-gation to publish the code. The policy of the license is a bit liberal, but it brings advantages,companies desire to do better and better products and they cooperate in development ofOpenCV library. OpenCV is used in many projects in the medical industry, security detec-tion, etc.

3.2 Qt Framework

Qt is a framework, which provides a cross-platform (Windows, Linux, Mac OS, etc.) userinterface creation. It allows to create graphical window objects whose look remain the samein several operating systems. It takes a component’s look directly of the operating system,so the application looks the same way anywhere.

Chapter 3. Analysis of Used Principles Ján Podmajerský

It was originally released in 1995 as a toolkit for C++ developers [Thelin, 2007], helpingthem to create a universal user interface. However nowadays it is available for a wide rangeof languages such as Java, Javascript, Ruby, Phyton and even .NET platform. Qt is actuallya set of objects which can be inherited and used in any project. It is based on the idea ofsignals and slots, which create communication between components. It is simple and canconnect buttons with actions and events. This graphic user interface programming method isnow used also in other graphic frameworks.

Qt is available under two licenses a commercial license, which is developed by Nokia and anopen source, which is created by Software Foundation. Open source does not require codesharing.

3.3 HSV Color Model

HSV stands for hue, saturation and value, it is a color model representing colors more natu-rally than RGB (red, green, blue), which is based on mixing those three basic colors to createa whole palette of colors.

Hue means the dominant color of the pixel. It is measured in angles on the circle colorpalette, where colors vary from red, through yellow, green, cyan, blue, and magenta, back tored. The scale is 0 – 360 degrees, every basic color have its own range angle, for instancegreen is at 120 – 180 degrees. Saturation is explained as a mix of white with dominant huecolor. If saturation is 0 it means there is only white without any mixture. An increasingnumber of saturation means more white added to the hue. The value describes the amountof dark or black color mixed with hue, it is similar to saturation, but when the value is 0 theresulting color is black. It is sometimes also called brightness, because it affects it in color.

Figure 3.1: HSV model cone1.

All these three parameters are depicted together in a cone (Figure 3.1). The cone is orientedwith its area of the base to the top [Levkowitz, 1997]. Hue is measured on this top circle. Thevalue is set on the vertical axis, values are increasing from bottom to the top. Saturation is onthe horizontal axis ranging in numbers from the center to the side. All those three parameters

1http://www.cis.temple.edu/~latecki/Courses/CIS581-02/MatCIS581-02/

12

http://www.cis.temple.edu/~latecki/Courses/CIS581-02/MatCIS581-02/


are mixed, but only saturation and value is affected by ambient light. As a result hue can beseparated, it allows to detect colors in different light conditions.

3.4 Homography

Homography is a transformation from one space to another. It can be used for attachingpanorama images, or for transforming points captured by a camera and in real world [Hart-ley and Zisserman, 2004]. Homography is estimated between images by finding featurecorrespondences in those images. Homography matrix H is defined as 3 × 3 and can beexpressed as a matrix multiplication. There is a point A [ xa, ya ], which is a point of objectO. Point B [ xb, yb ] is related to the same object O, but it is captured from different view(another coordinate system). The Homography H matrix is defined in (Equation 3.1).

xaya1

=

xa′/mya′/m

m/m

=

h11 h12 h13h21 h22 h23h31 h32 h33

∗xbyb1

(3.1)

sacos(φ) sbsin(φ) tzsasin(φ) sbcos(φ) typ0 p1 1

(3.2)

xa′ = h11x+ h12y + h13 (3.3)

ya′ = h21x+ h22y + h23 (3.4)

m = h31x+ h32y + h33 (3.5)

(xa′)/m− (h11x+ h12y + h13)/(h31x+ h32y + h33) = 0 (3.6)

(ya′)/m− (h21x+ h22y + h23)/(h31x+ h32y + h33) = 0 (3.7)

((xa′)/m) ∗ (h31x+ h32y + h33)− (h11x+ h12y + h13) = 0 (3.8)

((ya′)/m) ∗ (h31x+ h32y + h33)− (h21x+ h22y + h23) = 0 (3.9)

13


Ah =

−x −y −1 0 0 00 0 0− x −y −1...

∗

h11h12h13h21h22h23h31h32h33

= 0 (3.10)

Ah = 0, h 6= 0 (3.11)

Homography matrix encodes the geometric transformation in its parameters (Equation 3.2).φ is a roattion angle, s is scaling a, b is skewing and tz, ty are translations. Homographymatrix can be avaluated using homogeneous set of linear equations as stated before whichare defined in (Equation 3.10) and (Equation 3.11).

14

Chapter 4

Interactive Whiteboard Prototype

We designed a solution which will use LED pen for a detection, a video projector or displayand an off-the-shelf webcam for capturing the image from the projector in order to fulfillthe requirements. Our software will detect LED pen position by the webcam, afterwards itwill estimate the position of the LED and move mouse cursor to it. We need to be aware ofdifferent coordinate systems: the plane and the visual output unit system. Coordinates haveto be translated from one system to another. To estimate homography (Section 3.4), whichis used to translate between two planes (camera and computer desktop coordinate system),calibration process need to be done. This process begins with a chess board display, pointsof the chess board cornes are in the computer desktop system, while camera system capturesthem precisely. Our solution consists of following steps:

• calibration process using chessboard (Section 4.3),

• LED pen detection (Section 4.4),

• mouse cursor localization (Section 4.5).

4.1 Implementation Details

The position of the webcam may vary, the only restriction for positioning is to view thewhole projective plane (monitor). Left-handed people should locate on the right side ofthe projector, while right-handed on the left, in order to avoid shadowing the camera. Weused OpenCV (Section 3.1) functions for calibration process, finding chessboard in an im-age, computing homography and different thresholding methods, etc. We decided to useHSV color model, which provides the basis for image partial luminance invariance LED pensegmentation. Our application is implemented in Qt framework, which allows us to createmodern-looking software.

The coordinate systems have to be translated. The camera captures surrounding space aroundthe projective plane, there is an angle between camera view and projective plane. As a re-sult images captured by the camera cannot be directly used in the coordinate system of acomputer desktop. The left upper corner of the projected image has to be transformed to 0,0while being captured as xn, yn. The translation enables interaction in two different coordi-nate systems. To get the exact coordinates of the points in different systems it is importantto calibrate the camera. At first high-contrast black and white chessboard (Section 4.3) is

Chapter 4. Interactive Whiteboard Prototype Ján Podmajerský

displayed. It is captured by the camera and OpenCV (Section 3.1) algorithm thresholds theimage and finally finds the corner coordinates. The points from both coordinate systemsare assigned and a homography matrix can be estimated. The accuracy of the homographymatrix depends on a number of detecting points. Afterwards the application continues withLED pen detection and controlling the mouse cursor.

4.2 Camera Settings

The light conditions vary in every day application usage. Moreover cameras capture im-ages differently, it depends on the used chip, so camera settings vary on different cameras.We used Logitech B905 webcam, which provides maximal digital video resolution at 1280× 720 and captures in 30 fps. The main goal of our application is to detect LED pen inreal-time. In order to estimate images in real-time we adjusted default automatic camera set-tings, because of image processing algorithms relief, to detect LED pen more accurate. Weexamined different light conditions (Figure 4.1) and finally decided to change the settingsconstantly.

(a) Chessboard captured with optimized set-ting.

(b) Chessboard captured with similar settingthan in Figure a, but less exposure and lesscontrast. This is our final camera setting.

Figure 4.1: Images taken by different camera setting.

Different projecting devices require different settings. The optimal settings for monitor andprojector vary. The monitor is glossier, so it can behave like a mirror, while the projectoremits more light in the dark so a diode can reflect its light without being turned on. We had totake those factors into consideration to decide which settings would fit the most of commonconditions. Illustration of different settings can be seen on (Figure 4.2). If the image isprojected in a bright room, the captured image has to get dark to be able to capture LED pen.On the other hand if it is captured in a dark room, there is no need to darken the image, butstill it does not cause a remarkable difference but the LED pen have to be bright. As a resultwe set the parameters (Table 4.1). This settings fit to our camera, other products may havedifferent settings, but it does not have to be set precisely(5% approximation is acceptable).

16


parameter set value (in per cents) default value (in per cents)exposure 25 30gain 15 20contrast 25 20color intensity 40 20white balance 30 68brightness 58 40

Table 4.1: Camera setting.

(a) Camera is set to high gain andalso high contrast.

(b) Same setting, as in Figure a,but less contrast.

(c) Less gain than Figure b, it isthe optimal setting.

Figure 4.2: Images captured in the same light condition, thresholded with the same filter, butin different camera setting.

4.3 Automatic Calibration Process Using Chessboard

Before interacting with the projective plane it is important to determine its position in thecaptured image. It is estimated by automatic calibration. High-contrast (black and white)chess board is shown until it is detected by an image processing algorithm. Homography isestimated and finally an application can be used.

Figure 4.3: Detected chessboard in the original image in a dark room.

The chessboard is generated dynamically, dimensions are relative to the screen resolution(1366 × 768 means 6 × 3 points). The whole chess board has to be detected. Sometimeslight condition or the angle of the camera and projected image may cause problems. Thresh-olding methods are used to eliminate those problems. It is a method, which rejects pixels thatare less than entry value and those higher are strenghten, resulting in binary image creation.This image is optimal input for finding chess board by algorithm. However the whole image

17


has to be controlled to administer the threshold, it is a time consuming operation. there is apossibility to use the orginal image from the camera to find the calibration chessboard (Fig-ure 4.3), but we increased the detection rate by use of adaptive threshold filter. We testedvarious thresholding methods, and their settings.

(a) Chessboard captured by the camera. (b) Chessboard after binary threshold applied(the threshold value is 210).

Figure 4.4: Binary threshold.

We examined the images in different light conditions and it is impossible to find optimalbinary threshold (Figure 4.4). We could have seen adaptive filter is more precise than binary,it takes into account specific surrounding of each pixel.

(a) Image after adpative threshold filter withuse of Gaussian method.

(b) Image after adaptivethreshold filter withuse of Mean method. This particular settingis used in the final version with neighborhoodvalue at of 49.

Figure 4.5: Visual comparison of different methods of adaptive thresholds used with thesame image.

The adaptive threshold filter was tested to estimate the proper size of pixel neighborhood area(best chessboard detection rate in different light conditions). The size of a pixel neighbor-hood (Figure 4.6) is crucial to detect the chess board. We used adaptive mean (Figure 4.5(b))method: the value is the mean of the neighborhood of the pixel. Gaussian method (Fig-ure 4.5(a)) can be also used, but it it is slower and it takes into account more factors, studiesmore deeply pixel’s neighborhood, which is not necessary for finding chess board.

While testing threshold methods and their parameter, we analyzed different places with dif-ferent light conditions, sometimes they were not suitable for using video projector, because

18


(a) Image after adaptive threshold with neigh-borhood 107.

(b) Image after adaptive threshold with neigh-borhood 55.

Figure 4.6: Visual comparison of different neighborhoods on adaptive threshold filter withuse of Mean method.

of strong ambient light (Figure 4.8). As a result we are able to find chessboard even inextreme light conditions (Figure 4.7).

(a) Original image in a room during sunset. (b) Image after adaptive threshold.

Figure 4.7: Extreme conditions.

Chess board polygons size is an important feature of the detection. The algorithm looks forblack rectangles in a white field, the chessboard has to be surrounded by a white frame. Therewere few problems while using a computer monitor, because its rim is black. Using adap-tive threshold with high neighborhood parameter is not a solution (Figure 4.9(a)). Howeverwhen the parameter is decreased, using a projector with a high ambient light condition is notdetected. We decided to enlarge the border on the sides of the chess board (Figure 4.9(b)).

After the chessboard detection, corner points are used to estimate homography (Section 3.4).Afterwards LED pen detection can be done as stated in (Section 4.4).

4.4 LED Pen Detection

Interaction with the whiteboard is enabled by LED pen detection. One use LED installed ona pen to control the projective plane. Our software detects the position of the pen and callsthe mouse click event on this position. The LED pen simulates exactly the mouse left buttonbehavior, when the LED is on, click down is called, when it turns off the click is released.

19


(a) Original image with direct light incidenc-ing with the chessboard.

(b) Image after adaptive threshold, chess-board could not be found.

Figure 4.8: Different light conditions in the parts of the image.

(a) Chessboard with standard border. (b) Chessboard with adjusted border.

Figure 4.9: Visual comparison of different chessboards with different border size.

4.4.1 LED Pointer Construction

Green LED is used, because it is one of the basic colors and it is easier to segment. The diodeproduces light of 8000 mcd, it is high emitting light diode. Adjusted camera settings (Sec-tion 4.2) enable to track the diode efficiently. The voltage required for using the diode is3.3 V, we used 3 AA batteries, each having 1.5 V. As a result a resistor is necessary in theconstruction. We put all of these components together with power switch into the markertube (Figure 4.10). The construction of LED pen requires only basic electrotechnics abili-ties. The diode has to be put in the right direction and a resistor has to be mounted directlyto the diode, the switch has to be installed in energy supply. The expenditure for creating apen is about 2 euros.

4.4.2 Color Detection

Green color in the image cannot be simply thresholded, because there are many green-colored objects in a projection, for instance walls can be painted green, or a presenter canbe wearing a green shirt. The thresholding color has to be limited to LED color interval, butthere are many factors which affect the color of the diode in specific conditions. The mostinfluential is ambient light, this is the main reason why we decided to use HSV model (Sec-tion 3.3), which can be thresholded more intuitively. Hue is constant for every condition

20


Figure 4.10: The prototype of LED pen.

(LED is green), while saturation and value vary according to the ambient light. We exam-ined different conditions and various ambient light and found out, the saturation and valueare relative to the brightness and darkness of the whole image (Figure 4.11). If an image (thelight condition in a room where it is captured) is dark, the LED is more lucid and if an imageis bright, LED is brighter. Having taken into consideration all of this tests, we decided tochange the range of detection relatively to the ambient light conditions.

In order to investigate ambient light, the only possible way is to get the information directlyfrom the image. Average saturation and value are estimated from the whole image. If theaverage saturation is high and the average value is low, the image is dark, and vice-versa lowsaturation and a high value means dark image. This approximation is conditionally accu-rate, because the projective plane covers dominant part of the captured image, so the averagevalues are suitable for ambient light differentiate. While having these average values, thresh-olding interval can be set to find LED pen blob. There are s1, v1 – low interval endpoints ands2, v2 – high interval endpoints, s means saturation, v value. S means average saturation,while V average value. If the image is bright, judging from the tests, S is less then 130 andV is more than 100. When the image is dark S is more than 130 and V is more than 100.In daylight, usual presenting condition, S is more than 130 and V is less than 100. Else weset the constant interval: s1 to 109, s2 is 255, v1 195 and v2 255. This settings can be seenin (Table 4.2) .

endpoints S < 130 and V > 100 S > 130 and V > 100 S > 130 and V < 100s1 S * 0. 5 S / 1.5 S / 1.5v1 190 V * 1.2 V * 7s2 130 S * 1.2 S * 1.2v2 255 255 255

Table 4.2: Thresholding intervals.

The average values are controlled every 5 minutes, it would slow down the application if itis done more often. All in all the light conditions do not vary fast, so 5 minute interval issufficient. The interval values are used to threshold captured image, which returns a binaryimage with the white LED pen blob.

21


(a) Detection in the bright room.

(b) Detection in common light condition.

(c) Detection in the dark room.

Figure 4.11: Different conditions, different projector and different color of the same diodewith zoon within the LED pen positiion.

4.5 Cursor Position Localization

Localization of the mouse cursor is the most important of our solution. After retrieving thebinary image of LED pen detection, there is no single point detected, but more noise around,LED pen is detected as a dominant blob.

22


After analyzing binary images, we decided to create a histogram, which divides the imageinto rectangles in the scale of 10 pixels (Figure 4.12(a)). Every rectangle contains a specificamount of white pixels. The cursor is placed in the rectangle of the highest white pixel-rate.The final cursor position is set to an average value of the white pixels x and y coordinatesin this rectangle. The method enables interaction with the computer, but the inaccuracy isobvious when it comes to writing with the LED pen (Figure 4.12(b)) in well-known software(Microsoft Paint). The problem is caused by dividing the image. If the LED pen blob issituated in more histogram rectangles, the highest rate rectangle is only one, so the finalposition does not take into account all the blob. The solution would take into account moreneigbhbor regions around the highest peak, but it is difficult to decide how many regions areenough. Finally the administration could be slower operation than localization alone.

(a) Thresholded image of LED pen with his-togram mask on.

(b) Accuracy test.

Figure 4.12: Histogram cursor position method.

It would be more accurate if more points are taken into consideration. We experimented withusing all the image and the results improved. However the thresholding filter may detectthe surrounding of the projected screen, too. Sun may enlighten green objects, which fit theinterval, they are detected. Trees on the outside, were detected during the tests, which causedhigh inaccuracy. The thresholding interval for LED pen detection cannot be reduced, becauseLED would not be detected. However only LED detected in the projective plane has to betaken into consideration. Projective plane corner coordinates can be translated to the cameracoordinate system and finally the polygon (projective polygon) which has to be observedto localize the LED pen is created of those points in the captured image (Figure 4.13(b)).The polygon is divided into 2 triangles and is observed by filling triangle algorithm, which isused because of fast computation and high observing control. The final LED pen localizationis estimated at an average value of the coordinates of the whole projective polygon. Theaccuracy of the localization is higher, but still the average number of the blob does notprovide sufficient accuracy (Figure 4.13(c)).

The idea of reduced observation improves the detection, but the localization alone is notbetter. Average number of x and y coordinates in the projective polygon do not take intoconsideration the size of pixel groups, which creates small objects in the captured image.The centroid is more suitable for the localization, if white pixel blobs were objects in realworld the centroid would mean their physical centre, the point in the object, which is themost stable. In the estimation the position of all pixel groups are considered, not all thewhite pixels are treated the same in the captured image. The whole projective polygon isobserved by finding centroid algorithm implemented in OpenCV. This method provides the

23


(a) Original image captured by thewebcam.

(b) Detection of white pixels inprojective polygon.

(c) Accuracy test.

Figure 4.13: Average localization method.

best performance, the interaction is more accurate than previous. Writing is more conve-nient (Figure 4.14), but still there is a divergency within 5 pixels, which can be caused byinaccurate homography translation.

Figure 4.14: Smile paint using centroid method.

24

Chapter 5

Testing

We did usability test of our application by common users. We were aware of our detectioninaccuracy, the application works with precision about 5 pixels, but the tests should haveproved whether it is a problem for users. The test took place in a room with windows, therewas a daylight present. We used a video projector. The testers were on different computerskill level. The test was done by 9 people.

The testers should have worked intuitively with the solution, executed basic operations on acomputer. We explained them only the basic functionality of LED pen and they knew oursoftware should provide similar functionality to the interactive whiteboard.

Generally all the testers were impressed with the idea of the solution. Some of them alreadyworked with a commercial interactive whiteboard, and interaction with the computer of oursolution looks similar to them. Several problems occured with left mouse double click, theswitch in the LED pen is not sensitive enough, but also a standart operating system mousedoboule click time has to be increased. Other problems were with intended short mouseclicks, when users used the software keyboard, they tried to write a text, double letters werewritten in some cases, it was caused by the switch. The period of click has to be short, whichrequires experience with the prototype LED pen. The negative references were also aboutthe LED pen weight (3 AA batteries), which was only prototype solution for now.

The interaction with computer applications, web browser, picture viewer, satisfied all thetesters. However one tester, a student of IT, was not satisfied with the accuracy, he wantedto click on a small icon, instead an icon next to the desired was clicked. On the other handhe was surprised with the fast response. He tested the application more deeply and tried towrite a text in Microsoft Paint (Figure 5.1). When the application was tested by a woman,student of economics, who has never used an interactive whiteboard, we had to guide her toavoid shadowing the camera, to use the projective plane as a touch screen. She tended to usethe LED pen as a pointer from the distance, but the LED is diffuse, so the interaction wasnot successful. Finally she worked properly and appreciated the detection, but in Paint shenoticed the inaccuracy.

The tests showed the application can be used for computer interaction or using LED pen asa marker on a presentation, but when it comes to writing, the accuracy is not sufficient.

Tests were executed on our computer (Intel Core i5 – 430M, 2.27GHz processor and 4GBRAM) all processing worked in real-time. The average time needed for estimation one cap-tured image to localize the cursor was 24ms. It is less than 33 ms, so all the images from the

Chapter 5. Testing Ján Podmajerský

Figure 5.1: Image paint by a user.

camera, capturing at 30 fps, can be used.

26

Chapter 6

Conclusion

In this thesis we designed an inexpensive solution of interactive whiteboard using OpenCVand Qt library. After analyzing existing solutions, we found many restrictions for their usage.We were focused on creating interactive whiteboard, which could be used in common day-light conditions. We examined different light conditions. We did research in LED detection,we captured various images in different light conditions and analyzed them. We comparedsaturation and value in the HSV color model in those images and decided to set thresholdingintervals according to 3 basic light conditions. They are set according to the average satu-ration and value. In order to interact with the computer, it is important to place the mousecursor in the desired position precisely. We were not able to set the cursor with the precisionas commercial solutions have. Three methods of localization the cursor were compared, his-togram, average and centroid. The best results are provided by centroid method, which setsthe cursor in the centroid of the detected pixels in the projective polygon. The applicationcan work with video projector as well as monitor. Better results are provided using videoprojector, because plane is much bigger and there is no reflection.

Chapter 6. Conclusion Ján Podmajerský

28

Bibliography

Adrian Bradski. Learning OpenCV, [Computer Vision with OpenCV Library ; software thatsees]. O‘Reilly Media, 1. ed. edition, 2008. ISBN 0-596-51613-4. Gary Bradski andAdrian Kaehler.

Kelvin Cheng and Masahiro Takatsuka. Real-time monocular tracking of view frustumfor large screen human-computer interaction. In Proceedings of the Twenty-eighth Aus-tralasian conference on Computer Science - Volume 38, ACSC ’05, pages 125–133, Dar-linghurst, Australia, Australia, 2005. Australian Computer Society, Inc. ISBN 1-920-68220-1. URL http://dl.acm.org/citation.cfm?id=1082161.1082175.

R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. CambridgeUniversity Press, ISBN: 0521540518, second edition, 2004.

Kristian Jantz, Gerald Friedland, Lars Knipping, and Raúl Rojas. A low-cost mobilepointing and drawing device. In Proceedings of the international workshop on Educa-tional multimedia and multimedia education, Emme ’07, pages 121–122, New York, NY,USA, 2007. ACM. ISBN 978-1-59593-783-4. doi: 10.1145/1290144.1290165. URLhttp://doi.acm.org/10.1145/1290144.1290165.

Otto Korkalo and Petri Honkamaa. Construction and evaluation of multi-touch screens usingmultiple cameras located on the side of the display. In ACM International Conference onInteractive Tabletops and Surfaces, ITS ’10, pages 83–90, New York, NY, USA, 2010.ACM. ISBN 978-1-4503-0399-6. doi: 10.1145/1936652.1936667. URL http://doi.acm.org/10.1145/1936652.1936667.

Johnny Lee, Scott Hudson, and Pau Dietz. Hybrid infrared and visible light projection forlocation tracking. In Proceedings of the 20th annual ACM symposium on User interfacesoftware and technology, UIST ’07, pages 57–60, New York, NY, USA, 2007. ACM.ISBN 978-1-59593-679-0. doi: 10.1145/1294211.1294222. URL http://doi.acm.org/10.1145/1294211.1294222.

Johnny Chung Lee. Hacking the nintendo wii remote. IEEE Pervasive Computing, 7(3):39–45, 2008. ISSN 1536-1268. doi: http://doi.ieeecomputersociety.org/10.1109/MPRV.2008.53.

H. Levkowitz. Color theory and modeling for computer graphics, visualization, and mul-timedia applications[. Kluwer international series in engineering and computer science.Kluwer Academic Publishers, 1997. ISBN 9780585284286. URL http://books.google.sk/books?id=w7Zpk2wWywoC.

http://dl.acm.org/citation.cfm?id=1082161.1082175

http://doi.acm.org/10.1145/1290144.1290165

http://doi.acm.org/10.1145/1936652.1936667

http://doi.acm.org/10.1145/1936652.1936667

http://doi.acm.org/10.1145/1294211.1294222

http://doi.acm.org/10.1145/1294211.1294222

http://books.google.sk/books?id=w7Zpk2wWywoC

http://books.google.sk/books?id=w7Zpk2wWywoC

Bibliography Ján Podmajerský

Jiaqing Lin, Hiroaki Nishino, Tsuneo Kagawa, and Kouichi Utsumiya. Free hand inter-face for controlling applications based on wii remote ir sensor. In Proceedings of the9th ACM SIGGRAPH Conference on Virtual-Reality Continuum and its Applications inIndustry, VRCAI ’10, pages 139–142, New York, NY, USA, 2010. ACM. ISBN 978-1-4503-0459-7. doi: 10.1145/1900179.1900207. URL http://doi.acm.org/10.1145/1900179.1900207.

J. Thelin. Foundations of Qt Development. Expert’s Voice in Open Source. Apress,2007. ISBN 9781430202516. URL http://books.google.sk/books?id=nyxoEiGHKtoC.

30

http://doi.acm.org/10.1145/1900179.1900207

http://doi.acm.org/10.1145/1900179.1900207

http://books.google.sk/books?id=nyxoEiGHKtoC

http://books.google.sk/books?id=nyxoEiGHKtoC

Appendix A

Technical Documentation

Our solution is implemented in C++ using OpenCV image processing library and Qt frame-work. This technology enables to create real-time application. In this chapter we describemost important functions, the entire source code can be found on DVD.

We used several OpenCV functions these are the most important:

• findChessboardCorners(. . . ) – function finds chess board corners, used for calibrationprocess,

• findHomography(. . . ) – estimation of homography matrix,

• inRange(. . . ) – function used to threshold LED pen,

• moments(. . . ) – function used for estimation of centroid.

The most important function is the determination of HSV filter values and final threshold ofthe captured image (Listing A.1).

Listing A.1: HSV interval endpoints estimation.p = maths. data; //captured image in HSV color modelint satur = 0, val = 0;//saturation and valuesum = 0;

for(int i = 0; i < matHsv.rows*matHsv.cols; i++) {p++;//hsatur += *p++;//sval += *p++;//v++sum;

}

int Savg = satur/pocet, Vavg = val/pocet;//average saturation and value

if( Savg< 130 && Vavg > 100 ){//bright images1 = Savg/2;s2 = 130;v1 = 190 ;v2 = 255;

}else if( Savg > 130 && Vavg > 100 ){//dark images1 = Savg/1.5;s2 = Savg*1.2;//255if(s2 > 255) s2 = 255;v1 = Vavg*1.2;

Appendix A. Technical Documentation Ján Podmajerský

v2 = 255;}else if( Savg > 130 && Vavg < 100 ){/common conditions

s1 = Savg/1.5;s2 = Savg*1.2;//255if(s2 > 255) s2 = 255;v1 = Vavg*8;if(v1 > 200) v1 = 200;v2 = 255;

}

cv::inRange(matHsv,cv::Scalar(h1,s1,v1),cv::Scalar(h2,s2,v2),matHsv);

We implemented a function for the projective plane position determination in the capturedimage. It is based on a filling triangle algorithm. This function is called twice with projectiveplane corners, they are retrieved from the homography, for upper right triangle and lower lefttriangle of projective plane corner points. The plane is divided diagonally (Listing A.2).

Listing A.2: Function for exploring projective plane.int sum;//global variable for average coordinate valueint sumX, sumY; //variables for average coordinate value

bool Dialog::exploreTriangle(cv::Point p1, cv::Point p2, cv::Point p3,uchar *thresholded, uchar *newImage, int width){//width is the widthcv::Point a,b,c;//a min, b middle, c max, y coordinatesint i,j;float startX,endX,dx1,dx2,dx3;//dx1 a-b, dx2 a-c, dx3 b-cbool change = false;//whether there were any white pixels found//which y is the middle value,

if(p1.y > p2.y){if(p2.y > p3.y){

b = p2;c = p1;a = p3;

}else{if(p1.y > p3.y){

b = p3;c = p1;a = p2;

}else{b = p1;c = p3;a = p2;

}}


b = p1;c = p2;a = p3;


b = p3;c = p2;a = p1;

}else{b = p2;c = p3;

32


a = p1;}

}}

if (b.y-a.y != 0)dx1 = (float)((b.x-a.x))/(float)((b.y-a.y));

elsedx1 = 0;

if (c.y-a.y != 0)dx2 = (float)((c.x-a.x))/(float)((c.y-a.y));

elsedx2 = 0;

if (c.y-b.y != 0)dx3 = (float)((c.x-b.x))/(float)((c.y-b.y));

elsedx3 = 0;

if(b.x < a.x) {startX = a.x;endX = a.x;for(j = a.y; j < b.y; j++ ){//upper part

for(i = startX; i < endX; i++){if(thresholded[i+j*width] == 255){//we found the

white pixelnewImage[i+j*width] = 255;//set the new model

to find centroid++(histoX.at(i/10));++(histoY.at(j/10));change = true;

sumX += i;//for average cursor localizationsumY += j;//for average cursor localization++sum;

}}startX += dx1;endX += dx2;

}

startX = c.x;endX = c.x;for(j = c.y; j >= b.y; j-- ){//lower part


white pixelnewImage[i+j*width] = 255;//set the new

model to find centroid++(histoX.at(i/10));//for average cursor

localization++(histoY.at(j/10));//for average cursor

localizationchange = true;

sumX += i;sumY += j;++sum;

}

33


}startX -= dx3;endX -= dx2;}

}else{startX = a.x;endX = a.x;for(j = a.y; j < b.y; j++ ){//upper part


white pixelnewImage[i+j*width] = 255;//set the new model to

find centroid++(histoX.at(i/10));//for average cursor




}}startX += dx2;endX += dx1;

}

startX = c.x;endX = c.x;for(j = c.y; j >= b.y; j-- ){//lower part


white pixelnewImage[i+j*width] = 255;//set the new model to

find centroid++(histoX.at(i/10));//for average cursor




}}startX -= dx2;endX -= dx3;

}}

return change;}

34

Appendix B

User Guide

In order to use our application, these requirements have to be met:

1. Copy the whole solution source directory to your hard drive (from attaching DVD).

2. Turn on a webcam.

3. Adjust the camera settings.

Our prototype requires a 30 fps camera, to be able to detect images in-real time. Before usingthe software one have to adjust camera settings, as described in (Section 4.2). The camerahas to view the projective plane before starting the application.

Figure B.1: Example of user interface.

Application

After running the application, the dialog window appears, which provides 2 options (Fig-ure B.1). Basic mode is suitable for most of the users, it is a mode which provides the func-tionality of the whiteboard in silent mode, no information is displayed. This mode requires

Appendix B. User Guide Ján Podmajerský

the camera view being set in the projective plane. The chess board is displayed at the begin-ning, which calibrates coordinate systems. Afterwards the software interactive whiteboardcan be used. User can use LED pen to control any computer application. It is recommendedto use the application in minimized regime. If there is any problem advanced mode buttoncan be pushed, during the usage.

Advanced mode mode is an option for experienced users primary (Figure B.2). It displaysthresholding intervals, average saturation and value as well as the thresholded image. In thismode user can also evoke thresholding interval evaluation. There are 2 options: calibratebutton in order to display the chessboard and consequently use the application the same wayas in Basic mode. Adjust threshold is the option for estimation of the threshold interval, itcan be used if the light conditions change dramatically.

Figure B.2: User interface during the use.

36

Appendix C

Resumé

1 Úvod

S prezetáciami sa V dnešnej dobe casto stretávame. Sú súcast’ou prednášok na univerzitáchci vedeckých konferenciách. Prednášajúci sa snaží svojim výkladom posluchácov co najviaczaujat’. Používa pri tom rôzne pomôcky, ktoré mu pomáhajú spravit’ prednášku prít’ažlivoua zaujímavou. Historicky najstaršia klasická tabul’a patrila medzi najcastejšie. Má ale vel’anevýhod. Modernejšou pomockou môže byt’ tabul’a, na ktorú sa píše namiesto klasickejkriedy fixkami. Jej prednosti ocenia najmä alergici, prach z klasickej tabule môže byt’ prenich nebezpecný. V súcasnosti sa casto využívajú prostriedky, ktoré sú interaktívne. Video-projektor predstavuje kvalitatívne významný pokrok. Prezentujúci si pripraví svoj výklad apri prednáške ho premieta posluchácom. Pripravený výklad pozostáva z predspracovanýchdigitálnych poznámok, ktoré poskytne posluchácom. Tí si nemusia robit’ poznámky a môžusa tak naplno sústredit’ na prednášatel’ove slová. Ani v tomto riešení však nie je umož-nený prirodzený spôsob prezentovania. Nezachytáva doplnenia pripraveného výkladu o novépoznámky prednášajúceho, ktoré pripisuje k pôvodnému textu.

Interaktívna tabul’a predstavuje d’alší krok vo vývoji. Prednášajúci môže písat’ špeciálnymperom po vysielanej ploche, podobne ako písal na klasickú tabul’u. Okrem multimediálnehoobsahu zobrazeného na ploche sú všetky zmeny a doplnenia prezentácie uchované. Obstará-vacia cena multimediálnej tabul’e je v súcasnosti vysoká, zariadenie je neprenosné. Z týchtodôvodov nie je jej používanie vel’mi rozšírené.

Dostupné sú rôzne konfigurácie interaktívnej tabule. Jej súcast’ou je obvykle projektor ašpeciálna dotyková plocha. Táto špeciálna plocha je napríklad vytvorená na princípe pre-rušovania vedenia pri dotyku perom. Špeciálne elektormagnetické káble sú vedené v zadnejcasti plochy. Pri dotykoch pasívneho pera je porušené magnetické pole a tak sa dá urcit’pozícia pera. Toto riešenie je technicky znacne nárocné. Na vytvorenie interaktívnej tab-ule je možné použit’ aj jednoduchší princíp. Pri tomto riešení sa použijú bežne dostupnézariadenia. Projektor ako vysielac obrazu je použitý rovnako ako pri spomínanom riešení.Funkcia špeciálnej plochy a špeciálneho pera je nahradená kamerou, ktorá rozpoznáva svi-etiacu diódu umiestnenú vo svetelnom pere.

Appendix C. Resumé Ján Podmajerský

1.1 Požiadavky

Ciel’om tejto práce je predstavit’ vlastné riešenie interaktívnej tabule. Aby bolo toto navrho-vané riešenie prenosné a cenovo prijatel’né, je pri návrhu nutné použit’ vol’ne dostupné zari-adenia. Navrhované riešenie obsahuje nasledovné komponenty:

• webová kamera,

• video projektor,

• pero s vysokosvietivou diódou(LED) umiestenou na jeho špicke,

• softvér schopný detekovat’ v reálnom case pero s LED diódou.

2 Analýza dostupných riešení

Návrh na vytvorenie dostupnej verzie interaktívnej tabule bol predmetom viacerých prác.Viacerí navrhovatelia riešení dosiahli výborné výsledky. Tieto výsledky boli ale dosiahnutéiba vd’aka špeciálnym podmienkam použitia alebo vd’aka špeciálnym pomôckam, ktoré niesú bežne dostupné.

Jeden z prvých návrhov používa infracervené svetlo, ktoré rozpoznáva pomocou Wii Remote.Toto riešenie je presné, pretože Wii Remote používa infracervenú kameru. Ked’že infracer-vené svetlo je zložkou denného svetla, tak toto riešenie je použitel’né iba v miestnostiach bezprítomnosti priameho denného svetla.

Dalším prínosným návrhom riešenia problému je využitie klasickej kamery a zvýraznovaca,ktorý je dostatocne kontrastný až krikl’avý. Detekcia takéhoto zvýraznovaca je nepresná,nakol’ko projektor môže premietat’ na plochu farby rovnakého odtiena ako zvýraznovac.Oznacenie pozície (klikanie) je realizované pomerne neuspokojivým spôsobom. Používatel’podrží zvýraznovac urcitý cas na tej pozícii obrazu, na ktorú chce presunút’ ukazovatel’. Totoriešenie je nárocné na pozornost’ používatel’a a casto dochádza k nežiaducemu klikaniu.

Úplne iný prístup prináša riešenie, ktoré ponúka interakciu bez použitia ukazovacieho pera.Pri tomto návrhu sa na interakciu využívajú prsty používatel’a.Ten stojí celom k vysielanémuobrazu. Kamera je umiestnená nad obrazom. Používatel’ pracuje v tomto návrhu intu-itívne. Pozerá sa na miesto, kde chce presunút’ kurzor a zároven na toto miesto ukazujeukazovákom. Pri tomto riešení je problémom simulovanie kliknutia myši.

Po podrobnej analýze doterajších návrhov sme sa rozhodli, že návrh riešenia predkladanýv tejto práci bude využívat’ bežne dostupnú web kameru. Využije sa viditel’né svetelnéspektrum a bude rozpoznávat’ vysoko svietivú diódou. V návrhu využijeme rovnako akopri predchádzajúcich riešeniach projektor (monitor). Takáto konfigurácia nie je financnenárocná a mnohí používatelia už všetky potrebné komponenty vlastnia.

3 Analýza použitých princípov

Navrhované riešenie je naprogramované pomocou OpenCV, co umožnilo vytvorenie softvéruvykonávajúceho interakciu v reálnom case.Qt framework je použitý na vytvorenie grafického

38


vzhl’adu aplikácie. Na rozpoznanie LED pera je použité prahovanie, ktorému predchádzaprevod získaného obrazu z kamery do HSV farebného modelu.

V riešení sa pracuje v 2 súradnicových systémoch. Prvý je systém kamery, ktorá sníma LEDpero v priestore. Druhý je samotný súradnicový systém pocítaca. Na prevod medzi týmitosystémami sa používa homografia. Na vypocítanie homografnej matice sú potrebné dvo-jice jasne urcených bodov v týchto rozdielnych systémoch. Na získanie daných bodov bolpoužitý kalibracný proces, v ktorom sa zobrazí na celú vystielanú plochu šachovnica. Kam-era takýto objekt rozpozná a urcí súradnice rohov vo svojom systéme. V systéme pocítacasú tieto súradnice známe. Následne je vytvorená homografná matica.

3.1 Prototyp

Predkladaný návrh obsahuje naprogramovaný prototyp riešenia. Tento prototyp dosahujepresnost’ ovládania približne 5 pixlov. Toto riešenie je umožnené vd’aka kalibracnému pro-cesu. Na rozpoznanie šachovnice je potrebné najprv obraz z kamery predspracovat’. Napredspracovanie sa používa prispôsobené prahovanie. Bolo vykonaných viacero testov vrôznych podmienkach, aby sa dokázal urcit’ optimálny prahovací filter, ktorý je použitý zOpenCV knižnice. Štandardné nastavenie kamery bolo zmenené z dôvodu zjednodušeniadetekcie LED pera.

LED pero je z obrazu odprahované vzhl’adom na svetelné podmienky v obraze v case sníma-nia. Tieto podmienky sú skúmané každých 5 minút. Boli otestované hodnoty v HSV modelipri rôznych svetelných podmienkach a na základe toho bolo vykonané rozhodnutie pre urce-nie prahovacieho intervalu. Informácia o svetelných podmienkach bola získaná na základepriemernej hodnoty nasýtenia a hodnoty z HSV modelu. Následne bol podl’a týchto hodnôtobraz rozdelený do 3 kategórií: svetlý, tmavý, bežný. Podl’a týchto kategórií sa urcí hodnotavýsledného prahovacieho intervalu tak, že priemerné hodnoty z HSV modelu sa vynásobiarôznymi konštantami.

Na rozhodnutie umiestnenia pozície kurzora myši boli naprogramované 3 metódy. Prvározdelí binárny obraz z prahovania na mriežku o vel’kosti 10 pixlov. Vytvorí sa histogram,ktorý zist’uje pocet bielych bodov v jednotlivých políckach mriežky. Nakoniec sa z políckas maximálnym poctom bielych bodov vyráta priemerná hodnota zvlášt’ pre súradnice x a y.Kurzor sa presunie na túto pozíciu. Druhou metódou na presun kurzora je zohl’adnenie ibatej casti obrazu snímaného kamerou, ktorá je rozpoznaná ako obraz vysielaný projektorom.Obraz vysielaný projektorom je analyzovaný pomocou algoritmu vyplnania trojuholníkov. Ztejto plochy sa vyráta priemer súradníc všetkých bielych bodov. Na pozíciu priemeru súrad-níc sa umiestni kurzor. Druhá metóda sa ukazuje presnejšia ako prvá. Najlepšie výsledkyboli dosiahnuté tret’ou metódou. Pri tejto metóde sa namiesto priemeru vyráta t’ažisko sosúradníc bielych bodov.

4 Testovanie

Navrhované riešenie bolo otestované 9 l’ud’mi, ktorí boli na rôznej úrovni zrucnosti pri prácis pocítacom. Pri týchto testoch išlo o overenie, ci sa nepresnost’ odrazí na použitel’nosti.Všetci užívatelia sa zhodli na názore, že napriek nepresnosti je práca s pocítacom na pri-jatel’nej úrovni ovládania. Výnimkou bolo testovanie písania textu v nejakom grafickom

39


editore. Avšak na používanie pri prezentácii, na zvýraznovanie textu a posúvanie snímok jeriešenie postacujúce.

5 Záver

V tejto práci je implementovaný prototyp interaktívnej tabule, ktorá používa vol’ne dostupnékomponenty. Riešenie nie je statické, dá sa použit’ v bežných svetelných podmienkach, vakých sa používa projektor. Rozpoznanie LED pera nie je úplne presné, v urcitých situáciachvzniká nepresnost’ približne 5 pixlov.

40

Appendix D

DVD Contents

/source/ - source code of our prototypewith required libraries

/pdf/ - pdf version of this thesis/data/ - images captured by the camera while testing

Faculty of Informatics and Information Technologies -...

Documents

Transcript of Faculty of Informatics and Information Technologies -...