Human Computer Interaction Laboratory

@jonfroehlich Assistant Professor Computer Science

CMSC838 Tangible Interactive Computing

Week 11 | Lecture 19 | April 7, 2015 Depth Sensing and Microsoft Kinect

TODAY’S LEARNING GOALS

1. Learn about depth cameras and specifically the

Kinect v1 and Kinect v2

2. Learn how to download and use various Kinect v1

samples

3. Learn how to program a very simple Kinect app

that interacts with the Arduino (I will try to spend a

majority of the lecture on #3)

Quick Demo!

FADE AN LED BASED ON RIGHT HAND POSITION Created by Jon Froehlich for CMSC838f

Download this demo code on the MP4 webpage

Some Depth Camera Interactive Inspirations

INTERACTIVE PUPPET PROTOTYPE WITH XBOX KINECT Created by Design I/O using libfreenect Kinect drivers and ofxKinect

[source: https://vimeo.com/16985224]

PINOKIO LAMP Created by Shanshan Zhou, Adam Ben-Dror, Joss Doggett with Processing, Arduino, and OpenCV

[source: https://vimeo.com/52366512]

THE FORCE: CONTROLLING RC HELICOPTER WITH KINECT Created by CMSC838f Students Leyla Norooz and Darren Smith

[source: https://vimeo.com/87632677]

BE INSPIRED!

Spend time looking at the example

inspirations posted to the assignment

webpage

[source: http://cmsc838f-s15.wikispaces.com/MP4+Kinects+%26+Motors]

Depth cameras combine high-fidelity pixel mapped proximity

sensing with a standard RGB camera. Thus, the device delivers

not just RGB at each pixel position but depth as well.

DEPTH CAMERAS While the Microsoft Kinect is probably the most popular depth camera, there has been a veritable

explosion of depth sensing devices on the market in the past few years.

Creative Senz3D Depth and Gesture Recognition Camera

Microsoft Kinect v1 Microsoft Kinect v2

Google’s Project Tango

It used to be remarkable to have a camera built-in to your mobile phone. Now it would be hard to find a phone without a camera. I expect a similar trend for depth cameras in phones/tablets. It’s unclear how long this

progression will take but the progression itself seems inevitable

DEPTH SENSING IN EVERY PHONE?

Example: Google’s Project Tango

SAY HELLO TO PROJECT TANGO

[source: https://youtu.be/Qe10ExwzCqk]

Google ATAP

Take your Kinect out of the box. If there isn’t an HCIL

Hackerspace sticker on it, please affix one.

MICROSOFT KINECT VERSION 1 AND 2

Kinect for Windows v1 Kinect for Windows v2

Release Date Nov, 2010 (Xbox 360); Feb, 2012 (Windows) Nov, 2013 (Xbox One); July, 2014 (Windows)

Color Camera 640 x 480 @ 30 fps 1920 x 1080 @ 30fps

Depth Camera 320 x 240 @ 30 fps 512x424 @ 30fps

Range 0.4 – 4.5 meters (1.3 – 14.76 feet) 0.5 – 4.5 meters (1.6 – 14.76 feet)

Horizontal FOV 57 degrees 70 degrees

Vertical FOV 43 degrees 60 degrees

Tilt Motor Yes No

Skeleton Joints 20 joints 25 joints

Number of Skeletons 2 6

Microphones 4 microphones 4 microphones

[source: https://www.microsoft.com/en-us/kinectforwindows/meetkinect/features.aspx; http://channel9.msdn.com/coding4fun/kinect/Kinect-1-vs-Kinect-2-a-side-by-side-reference]

MICROSOFT KINECT V1

HOW THE KINECT DEPTH SENSOR WORKS

[source: https://youtu.be/uq9SEJxZiUg]

LEARN MORE ABOUT HOW KINECT V1 WORKS Microsoft Research Keynote

[source: http://videolectures.net/ecmlpkdd2011_bishop_embracing/]

KINECT V1 TEARDOWN

[source: https://www.ifixit.com/Teardown/Microsoft+Kinect+Teardown/4066]

ifixit.com

MICROSOFT KINECT V2

[source: https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725]

KINECT V1 VS. KINECT V2

[source: https://youtu.be/eNIP9nFo9n4]

MICROSOFT KINECT V2

[source: https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725]

MICROSOFT KINECT V2

[source: https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725; http://ifixit.org/blog/5684/our-first-look-at-the-xbox-one-kinect-ir-field/]

MICROSOFT KINECT V1 VS. V2 INFRARED The Microsoft Kinect v1 and v2 have very different depth sensing technology

The Microsoft Kinect v1 emits a unique infrared pattern that is used to localize objects by a second infrared camera

The Microsoft Kinect v2 has three infrared emitters that pulse infrared into the environment. The v2 infared sensor then measures the reflection time (time of flight).

[source: https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725; http://ifixit.org/blog/5684/our-first-look-at-the-xbox-one-kinect-ir-field/]

MICROSOFT KINECT V2 INFRARED DEPTH SENSING TECH The Kinect v2 measures the time between an outgoing infrared light pulse and a corresponding

incoming light pulse to measure physical distance

[source: http://ifixit.org/blog/5684/our-first-look-at-the-xbox-one-kinect-ir-field/; http://blogs.microsoft.com/blog/2013/10/02/collaboration-expertise-produce-enhanced-sensing-in-xbox-one/;

http://www.gamasutra.com/blogs/DanielLau/20131127/205820/The_Science_Behind_Kinects_or_Kinect_10_versus_20.php]

[0055] The image camera component may include an IR light component, a three-dimensional (3-D) camera, and an RGB camera that may be used to capture the depth image of a capture area… In some embodiments, pulsed infrared light may be used such that the time between an outgoing light pulse and a corresponding incoming light pulse may be measured and used to determine a physical distance from the capture device to a particular location on the targets or objects in the capture area… The phase of the outgoing light wave may be compared to the phase of the incoming light wave to determine a phase shift. The phase shift may then be used to determine a physical distance from the capture device to a particular location on the targets or objects.

Source: http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.html&r=1&f=G&l=50&s1=%2220120278904%22.PGNR.&OS=DN/20120278904&RS=DN/20120278904

LEARN MORE ABOUT WHAT’S INSIDE KINECT V2 Jamie and Adam: Tested

[source: https://youtu.be/-RELJD0LBg8]

INSIDE THE BRAINS OF XBOX ONE KINECT Microsoft Research

[source: https://youtu.be/ziXflemQr3A]

[source: http://people.csail.mit.edu/mrub/vidmag/]

KINECT V2 TEARDOWN

[source: https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725]

ifixit.com

Developing for Kinect v2

DEVELOPING FOR KINECT V2

[source: http://cmsc838f-s15.wikispaces.com/MP4+Kinects+%26+Motors]

Following the directions on the

assignment webpage

MAKE SURE YOU HAVE USB 3.0-CAPABLE HARDWARE

[source: http://blogs.msdn.com/b/usbcoreblog/archive/2012/06/27/how-to-determine-whether-a-usb-3-0-device-is-operating-at-superspeed.aspx]

Download and run the Kinect

Configuration Verifier

http://go.microsoft.com/fwlink/?LinkID=513889

http://go.microsoft.com/fwlink/?LinkID=513889

SEE MICROSOFT KINECT V2 TROUBLESHOOTING GUIDE

[source: https://social.msdn.microsoft.com/Forums/en-US/20dbadae-dcee-406a-b66f-a182d76cea3b/troubleshooting-and-common-issues-guide?forum=kinectv2sdk]

SEE ALSO 2.0 SDK DEVELOPER KNOWN ISSUES

[source: https://msdn.microsoft.com/en-us/library/dn758677.aspx]

OFFICIAL MICROSOFT KINECT V2 HOW-TO VIDEOS

[source: https://www.microsoft.com/en-us/kinectforwindows/develop/how-to-videos.aspx]

Developing for Kinect v1: Developer Toolkit Browser

DEVELOPING FOR KINECT V1

[source: http://cmsc838f-s15.wikispaces.com/MP4+Kinects+%26+Motors]

Open the Developer Toolkit Browswer v1.8

KINECT EXPLORER Kinect for Windows Developer Toolkit v1.8.0

The built-in servo motor provides programmatic control of tilt

angle. The built-in accelerometer checks to make sure the Kinect is

parallel to ground and its current tilt angle.

KINECT EXPLORER Kinect for Windows Developer Toolkit v1.8.0

The Kinect uses machine learning to recognize bodies and infer position. This is not always accurate.

DEPTH BASICS Kinect for Windows Developer Toolkit v1.8.0

With near mode enabled, the Kinect for Windows sensor can

accurately sense 0.4 meters (1.3 feet)

AVATEERING Kinect for Windows Developer Toolkit v1.8.0

Click SDKs and install XNA Game Studio 4.0

Open the Avateering-XNA.sln file in the newly installed Avateering directory

Click the ‘Play’ button to launch in debug mode

Unlike the other examples, you cannot run Avateering directly from the Developer Toolkit

window. Instead, you must install the source and run the code from Visual Studio. Note:

Avateering requires XNA Game Studio 4.0, so if you don’t have this installed—do that first

(as shown above)!

3D-RECONSTRUCTED HEAD NIGHT LIGHT? For IA3, maybe use Kinect to scan an object and turn it into a night light?

Developing for Kinect + Arduino

Recommended approach to MP4: browse and play with the

Microsoft sample code in the Developer Toolkit—use it to inspire

your ideas and then remix this code into a working solution for

your interaction with the Arduino

The code from class today is posted to the MP4 website and Piazza