Increasing Kinect Application Development Productivityby an Enhanced Hardware Abstraction

Bernardo Reis (bfrs@cin.ufpe.br)

Virtual Reality and Multimedia Research Group

Federal University of Pernambuco, Computer Science Center

João Marcelo Teixeira (jmxnt@cin.ufpe.br)

Felipe Breyer (fbb@cin.ufpe.br)

Luis Arthur Vasconcelos (lalv@cin.ufpe.br)

Aline Cavalcanti (asc3@cin.ufpe.br)

André Ferreira (amf2@cin.ufpe.br)

Judith Kelner (jk@cin.ufpe.br)

Development Difficulties

Necessity to have enough space

• At least 2m from the device

Physical weariness during tests

• Due to repetition

Specific knowledge for the right movements

• Necessity to have the support of a specialist during development and testing

Necessity to own the accessory

• One for each developer

2/19

Solution

A tool that enables playback of previously recorded information from Kinect (as if being streamed directly

from the device), supplying the developer with sufficient mechanisms for a productive development environment.

3/19

Existing Solutions

Cons

Can’t access skeleton data

Can’t perform depth/color registration

Based on a very low level library

Pros

Open-source

Avoids changes in code

4/19

(FAKENECT)

Existing Solutions

Cons

No access to audio or elevation motor

Changes in code are required

Pros

Open-source

Performs depth/color registration

Can record while running the user application

5/19

(BUILT-IN)

Existing Solutions

Cons

Requires to have a Kinect connected

Pros

Saves the dump transparently

Access directly the driver

Can manipulate the input at runtime

6/19

(KINECT STUDIO)

Identified Problems

Good specific solutions, such as:

• No changes in code

• Record while running user application

But they all lack a sense of a complete and productive environment, missing important features that we believe are important.

7/19

KINA Toolkit

A toolkit for reducing the effort of the programmer on the development of body-tracking based applications that make use of Microsoft Kinect SDK. Its design principles are:

Clean and simple API

Require the minimum possible knowledge

Support a productive development model

8/19

KINA Toolkit

9/19

Recorders

Simple Recorder

• Small application for recording Kinect data

• Independent from the user application

• Should be used for initial specifications of movements

Pass-through Recorder

• Library that intercepts user streams and records them

• Enables feedback from the user application

• Should be used for later testing

10/19

Recorders

Very low storage overhead!

Can store:

• RGB

• Depth/Depth + Player

• Skeleton

Compressed:

• JPEG (color), 90% quality

• PNG (depth)

• 93% reduction in size

11/19

Movement Database

12

Editor

Allows cropping of dump files

• Adjusting them to your needs

• Generating multiple dumps from one record

• In the future, will also combine dumps

13/19

Emulator

Playback mechanism of KINA

A C++ library that can be used without any change in code

Maintains temporal coherence with the original input

Currently, only supports pooling mode of capturing/reading

Small RAM overhead (11MB against 158MB of the MS SDK)

14/19

Development Model

15/19

In Advanced Topics in Media and Interaction class:

• 12 CS students, 20 to 27 years old, 1 woman

• No previous knowledge of natural interaction

• 3 groups, based on acquaintance between them

Experiment

Functional training program Art experiment wherethe body is a brush

Bug-smashing game

16/19

Experiment Feedback

Suggestions:

• Possibility to dynamically change the dump during execution

• GUI for the simple recorder

“Indispensable during test phase”

“it reduced the overall time of development”

17/19

The toolkit was positively evaluated by the students.

It is essential to have similar solutions for multi-dimensional interface devices.

The development model was shown to be well suited for Kinect based applications.

Conclusions and Future Works

Future work:

• Combining dumps in the editor

• Event-based playback support

• C# API

18/19

Increasing Kinect Application Development Productivityby an Enhanced Hardware Abstraction

Bernardo Reis (bfrs@cin.ufpe.br)

Virtual Reality and Multimedia Research Group

Federal University of Pernambuco, Computer Science Center

João Marcelo Teixeira (jmxnt@cin.ufpe.br)

Felipe Breyer (fbb@cin.ufpe.br)

Luis Arthur Vasconcelos (lalv@cin.ufpe.br)

Aline Cavalcanti (asc3@cin.ufpe.br)

André Ferreira (amf2@cin.ufpe.br)

Judith Kelner (jk@cin.ufpe.br)