Programming with kinect v2
-
Upload
matteo-valoriani -
Category
Technology
-
view
176 -
download
0
Transcript of Programming with kinect v2
Matteo Valoriani
Nice to Meet You
Matteo Valoriani
PhD at Politecnico of Milano
CEO of Fifth Element
Speaker and Consultant
Microsoft MVP for Kinect
Intel Software Innovator: RealSense
email: [email protected]
twitter: @MatteoValoriani
linkedin: https://it.linkedin.com/in/matteovaloriani
2
Usage Scenarios
TRAININGRETAIL HEALTHCARETHERAPY EDUCATION
You have to be a
magician…
… or at least a
good illusionist
“Any sufficiently
advanced
technology is
indistinguishable
from magic”
(Arthur C. Clarke)
Sensor Anatomy
Kinect 2 - Specs
3D DEPTH SENSOR
RGB CAMERA
MULTI-ARRAY MIC
Hardware:
Depth resolution:512×424
RGB resolution:1920×1080 (16:9)
FrameRate:60 FPS
Latency:60 ms
Kinect Adapter
Kinect 2 vs Kinect 1
Version 1 Version 2
Depth range 0.4m → 4.0m 0.4m → 4.5m
Color stream 640×480 1920×1080
Depth stream 320×240 512×424
Infrared stream None 512×424
Type of Light Light coding ToF
Audio stream 4-mic array 16 kHz 4-mic array 48 kHz
USB 2.0 3.0
# Bodies Traked 2 (+4) 6
# Joints 20 25
Hand Traking External tools Yes
Face Traking Yes Yes+Expressions
FOV 57° H 43° V 70° H 60° V
Tilt Motorized Manual
System / Software Requirements
OS Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64)
CPU Intel Core i7 3.1GHz (or higher)
RAM 4GB (or more)
GPU DirectX 11 supported
USB USB 3.0
Compiler Visual Studio 2012, 2013 (Supported Express)
Language Native (C++), Managed (C#,VB.NET), WinRT (C#,HTML)
Other Unity Pro (Add-in), Cinder, openFrameworks (wrapper)
DEMOKinect Evolution
Accessing Kinect Data Sources
High-level architecture
Basic Flow of Programming
Sensor Stream Frame Data
Sensor Source Reader Frame Data
Kinect for Windows SDK v1
Kinect for Windows SDK v2
Source independent to each Data(e.g. ColorSource, DepthSource, InfraredSource, BodyIndexSource, BodySource, …)
Doesn’t depend on each other Source(e.g. Doesn't need to Depth Source when retrieve Body Data)
• In “New Project” create a new Windows Store app
• Enable Microphone and Webcam capabilities
• Add a reference to Microsoft.Kinect
• Use the Microsoft.Kinect namespace in your code
Creating a new store app using Kinect
Represents a single physical sensor
Always valid: when device is disconnected no more frame are generated.
Use IsAviable Property to verify if the device is connected
The KinectSensor class
this KinectSensorthis// Make the world a better place with Kinectthis
Sources
Give access to frames– Events
– Polling
Multiple readers may be
created on a single source
Readers can be paused
Readers
InfraredFrameReader reader = sensor.InfraredFrameSource.OpenReader();
reader.FrameArrived += InfraredReaderFrameArrived;
...
Frame references
void InfraredFrameReaderInfraredFrameArrivedEventArgs
using (InfraredFrame frame = args.FrameReference.AcquireFrame())
{if (frame != null){
// Get what you need from the frame}
}}
Sent in frame event args
AcquireFrame gives access to the actual frame
• Gives access to the frame data– Make a local copy or access the underlying buffer directly
• Contains metadata for the frame– e.g. Color: format, width, height, etc.
• Important: Minimize how long you hold onto the frame– Not Disposing frames will cause you to not receive more frames
Frames
• Allows the app to get a matched set of frames from multiple
sources on a single event
• Delivers frames at the lowest FPS of the selected sources
MultiSourceFrameReader
MultiSourceFrameReader MultiReader =Sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color |
FrameSourceTypes.BodyIndex |FrameSourceTypes.Body);
var frame = args.FrameReference.AcquireFrame(); if (frame != null) {
using (colorFrame = frame.ColorFrameReference.AcquireFrame())using (bodyFrame = frame.BodyFrameReference.AcquireFrame())using (bodyIndexFrame = frame.BodyIndexFrameReference.AcquireFrame()){
//}
}
System
Kinect
Driver
Kinect SDK
Application
Kinect
Driver
Kinect SDK
Application
Kinect
Service
Kinect SDK
Application
Kinect SDK
Application
Kinect for Windows v1 Kinect for Windows v2
Connection
Kinect for Windows v1 Kinect for Windows v2
PCPC
Example Multiple Connection
PCPC PC
Hub
Server
DEMODisplay frame in a Windows Store App
Data Source Details
Physical image sensors, ColorFrameSource,
InfraredFrameSource, DepthFrameSource,
BodyIndexFrameSource, BodyFrameSource, AudioFrameSource
Kinect Data Sources
• 30 or 15 fps, based on lighting
conditions
ColorFrameSource
InfraredFrameSource
– 16-bit distance in millimeters from the
sensor’s focal plane
DepthFrameSource
– 0 to 5: Index of the corresponding body,
as tracked by the body source
– > 5: No tracked body at that pixel
BodyIndexFrameSource
255
0 1
• Range is 0.5-4.5 meters
• Frame data is a collection of Body objects each
with 25 joints– Each joint has position in 3D space and an
orientation
• Up to six simultaneous bodies
• 30fps
• Hand State on 2 bodies
• Lean
BodyFrameSource
Skeleton vs Body
Coordinate System
ColorSpace (Coordinate System of the Color
Image)
– … Color
DepthSpace (Coordinate System of the
Depth Data)
– … Depth, Infrared, BodyIndex
CameraSpace (Coordinate System with the
origin located the Depth Sensor)
– … Body (Joint)
• Three coordinate systems
• Coordinate mapper provides conversions between each system
• Convert single or multiple points
Coordinate mapping
Name Applies to Dimensions Units Range Origin
ColorSpacePoint Color 2 pixels 1920x1080 Top left corner
DepthSpacePoint Depth,
Infrared,
Body index
2 pixels 512x424 Top left corner
CameraSpacePoint Body 3 meters – Infrared/depth
camera
DEMOHandle body frames and coordinate mapping
Kinect Studio
Recording and Playback
Recordable Data Sources
Infrared
13 MB/s
Depth
13 MB/s
BodyFrame
BodyIndex
Color
120 MB/s
Audio
32 KB/s
Legend
Record/Play
Record Only
DEMOKinect Studio 2.0Recording | Playback
Gesture Recognition
Heuristic + Machine Learning Approaches
DEMOProgrammatic detection of a gesture/pose
Gesture Builder
• New tool, shipping with v2 SDK
• Organize data using projects and solutions
• Give meaning to data by tagging gestures
• Build gestures using machine learning technology– Adaptive Boosting (AdaBoost) Trigger
• Determines if player is performing gesture
– Random Forest Regression (RFR) Progress
• Determines the progress of the gesture performed by player
• Analyze / test the results of gesture detection
• Live preview of results
Visual Gesture Builder
Your Application
DEMOMachine Learning Demo
Heuristic
• Gesture is a coding problem
• Quick to do simple
gestures/poses (hand over head)
• ML can also be useful to find
good signals for Heuristic
approach
Machine Learning (ML) with G.B.
• Gesture is a data problem
• Signals which may not be easily
human understandable (progress
in a baseball swing)
• Large investment for production
• Danger of over-fitting, causes you
to be too specific – eliminating
recognition of generic cases
Gesture Recognition
Hand Pointer Gestures
Hand pointer gestures
Engagement Targeting
Press
Panning/Zoom
DEMOControls Basics – XAMLUserViewer | Engagement | Targeting | Press
DemoBuilding ControlsBasics-XAML from Scratch
• File New Project > Grid App
• Add Reference to Microsoft.Kinect.Xaml.Controls
• Enable Microphone + Camera capabilities for app
• Add KinectRegion as container for rootFrame in
App.xaml.cs
• Run it!
Window
Frame
Page
Window
Frame
Page
Grid
KinectRegionKinect
UserViewer
Adding Hand Cursor Support
System Engagement Manual Engagement
• Hands over head
• In certain location of room
• Etc…
“Engagement” for hand cursor
KinectRegion.SetOnePersonSystemEngagement()KinectRegion.SetTwoPersonSystemEngagement()
KinectRegion.SetOnePersonManualEngagement(…)KinectRegion.SetTwoPersonManualEngagement(…)
DEMOPHiZ (Physical Interaction Zone) & KinectPointerPoint
PHIZ – Physical Interaction Zone
X / Y - For Cursor Rendering
KinectPointerPoint->Position->X– Between 0.0 and 1.0
KinectPointerPoint->Position->Y– Between 0.0 and 1.0
X / Y – Allows offscreen info
KinectPointerPoint->Properties->UnclampedPosition->X
KinectPointerPoint->Properties->UnclampedPosition->Y
Z
KinectPointerPoint->Properties->HandReachExtent (raw)
KinectPointerPoint->Properties->PressExtent (affected by user/UI)
Correlating PointerPoint to NUI Data
KinectPointerPoint->Properties->BodyTrackingId
KinectPointerPoint->Properties->HandType
KinectPointerPoint->Properties->BodyTimeCounter
Press
Recommended Minimum Size:• 208 x 208 (in 1080p resolution)• Press attraction towards center
• Larger buttons will just attract away from the edge
Adapting to smaller visual sizes:• Make large-size hit testable• Set KinectRegion.KinectPressInset
(Thickness) to non-visible part of button• Attraction region is smaller
Cursors
DemoControls Basics – XAML
Scroll
Zoom
Interacting with Controls (in Preview release)
• UI Frameworks– XAML (Store)
– WPF
• DirectX (Store)ControlsBasics-
XAML
Microsoft.Kinect.Xaml.Controls
Microsoft.Kinect.Toolkit.Input
WindowsPreview.Kinect
ControlsBasics-WPF
Microsoft.Kinect.Wpf.Controls
Microsoft.Kinect
ControlsBasics-DX
KinectCoreWindow.PointerMoved,KinectPointerPoint, KinectGestureRecognizer
InputPointerManager, PressableModel, ManipulatableModel
Controls Layer – Cursor rendering, Hit Test, Route PointerPoint via InputPointerManager
App (host KinectRegion, etc…)
Interacting with Content
• Navigating a camera through a 3d space
• Rotating a 3d object
• “Scrubbing” a video
Kinect Resources
• General Info & Blog -> http://kinectforwindows.com
• Purchase Sensor -> http://aka.ms/k4wv2purchase
• v2 Preview SDK -> http://aka.ms/k4wv2previewsdk
• Developer Forums -> http://aka.ms/k4wv2forum
Q&A
Tutto il nateriale di questa sessione sul sito
http://www.dotnetcampus.it
@MatteoValoriani
©2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Office, Azure, System Center, Dynamics and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.