Kinect with People and the Real WorldAndreas Erben

WIN-B210

Session Objectives And TakeawaysSession Objective(s):- Introduce Kinect for Windows V2- Understand capabilities of K4W V2

Key takeaways:- Have an understanding what Kinect for

Windows V2 can do for you- Get pointers to get started

4

• daenet was established in 1998• Gold Certified Partner• WPC Partner of the Year 2005, 2010• Regular “Inner Circle” / “Core Partner”

with Microsoft • Featured in joint Microsoft-Gartner

newsletter and in Gartner Magic Quadrant

Andreas Erben, daenet

• More than 15 years experience in the “Integration” and architecture space

• V-TSP Integration• Consulting, Coaching,

Implementation, Outsourcing• Scrum Master, Trainer, Coach• Passionate about Innovation• Community, User Group,

Presentations

• http://HackAndi.com• v-anerbe@microsoft.com• andreas.erben@daenet.com

Geos:Western Europe (Germany-HQ)Americas (USA)South-eastern Europe

http://www.daenet.com

Overview

Welcome to Kinect• A new kind of Camera, that can see in

3D• A traditional video camera• A 4-element microphone array• A powerful SDK which can:• Recognize that a person is in front of the sensor, and what position their body is in

• A software controllable directional microphone, and speech recognition technology

• And lots more

Wider field of view (horizontally and vertically) for depth and color3x depth fidelity1080p, HD color camera (30fps)Lighting independent IR (30 fps)Improved microphones (zero balanced)25 skeletal joints for six people -> Bio correct skeletal joints (particularly hips/shoulders/spine) plus more points and rotationsThumb tracking, end of hand tracking, Open/ClosedImproved range -> .5M near, 4.5M far (extended depth to 8M)

Kinect for Windows v2 Sensor Features

New sensor shipping now http://aka.ms/k4wv2purchase

New v2 SDK (public now) http://aka.ms/k4wv2sdk

Single set of cross-platform Kinect APIs (Desktop, Store, Xbox)

Create & publish Kinect apps to the Windows Store Still have robust support for WPF & Windows desktop applications

Kinect Unity plugin for both desktop & store apps

What’s New in K4W v2

Usage ScenariosTRAININGRETAIL HEALTHCARETHERAPY EDUCATION

Sensor Components

Power Light

RGB Camera

IR Emitters

Microphone Array

https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725

Kinect for Windows V2

Color (1920x1080) at 30 FPS (15 FPS low light)

Infrared (512x424) @30 FPS

Depth (512x424) @30 FPS

Body-index pixel-mask at 512x424 @30 FPSPerson detection and

joint tracking, up to 6 people @30 FPS

Floor plane detection

Transformation between coordinate systems

Event driven and polling-based APIs

J. Sell and P. O’Connor, “The Xbox One System on a Chip and Kinect Sensor,” IEEE Micro, 2014

Kinect Data Sources

Audio

Color Infrared Depth

BodyIndex

Body

High-level architecture

Kinect Drivers

Kinect Runtime

.NetAPI

.NetApps

Native API

Native Apps

WinRT API

Windows Store AppsC#, VB, JS, C++/CX

Physical Kinect Sensor

CPU -> i7 (recommended)RAM -> 4GB (recommended)GPU -> DirectX 11 (required)USB 3.0 -> Intel or Renesas chipsets (required)OS -> Windows 8.0/8.1 (required)

Recommended Hardware

Minimize per-frame allocationsLow latency – high throughputExpose low level data and high level interpretations

Design Principles

Kinect Data Sources

• Color Sensor• 1920 x 1080 • 30 or 15 fps, based on lighting

conditions

• IR Sensor• 512 x 424• 30 fps

Two Physical “Image” Sensors

• 1920 x 1080 array of color pixels• 30 or 15 fps, based on lighting

conditions

• Mirrored image to match other formats

ColorFrameSource

• Pixel Data• 16-bit IR intensity value

• Ambient light removed

InfraredFrameSource

• Range: 0.5 - 8 meters• Pixel Data

• 16-bit distance in millimeters from the sensor’s focal plane

DepthFrameSource

• Pixel Data • 0 to 5: Index of the corresponding

body, as tracked by the body source• > 5: No tracked body at that pixel

BodyIndexFrameSource

Range is 0.5-4.5 metersFrame data is a collection of Body objects each with 25 joints

Each joint has position in 3D space and an orientation

Up to six simultaneous bodies30fpsHand State on 2 bodiesLean

BodyFrameSource

Data is audio samples captured over a specific interval of timeAudio data is associated with an “audio beam”

A steerable “cone” of focus for audioMay be automatically or manually aimed

AudioFrameSource

Demo

Kinect SDK and Kinect Evolution (DataSources)

Interactions and Speech

Hand pointer gestures

Engagement Targeting

Press

Panning/ZoomAvailable to desktop or store apps as an Opt-In input mechanism with Kinect for

Windows 2.0

Speech Engagement for Global

Commands

Available to desktop apps (ONLY) as an Opt-In input mechanism with Kinect for

Windows 2.0

“Xbox”

Engagement forApp Commands

“Xbox Select”

Xbox Onew/ Kinect

Windows 8.0 or 8.1

w/ Kinect

N/A“Kinect”

or …

“In Experience” Commands

App-defined

grammar

App-defined

grammar

Demo

Interactions

Kinect Studio and Gesture Recognition

Recordable Data SourcesInfrared13 MB/s

Depth13 MB/s

BodyFrame

BodyIndex

Color120 MB/s

Audio32 KB/s

Legend

Record/Play

Record Only

V2.0Now can record AudioPlayback works without sensor attachedAPI access to playback/record

Kinect Studio, only uses these public APIs!

Kinect Studio – Major Improvements

Heuristic

Gesture is a coding problemQuick to do simple gestures/poses (hand over head)ML can also be useful to find good signals for Heuristic approach

Machine Learning (ML) with G.B.

Gesture is a data problemSignals which may not be easily human understandable (progress in a baseball swing)Large investment for productionDanger of over-fitting, causes you to be too specific – eliminating recognition of generic cases

Gesture Recognition

Posture recognition – heuristic/algorithmic

Floor normal

Upper legvector

Angle between Floornormal and bone

Common sense assumptions about posture:• Spine less than 45˚ from

floor normal• Upper legs less than 45˚

from floor normal

Standing

• Spine less than 45˚ from floor normal

• Upper legs more than 45˚ from floor normal

Sitting

• Spine more than 45˚ from floor normal

• Upper legs more than 45˚ from floor normal

Lying

Gesture Builder

New tool, shipping with v2 SDKOrganize data using projects and solutionsGive meaning to data by tagging gesturesBuild gestures using machine learning technology

Adaptive Boosting (AdaBoost) TriggerDetermines if player is performing gesture

Random Forest Regression (RFR) ProgressDetermines the progress of the gesture performed by player

Analyze / test the results of gesture detectionLive preview of results

! Machine learning innovations for your apps for free !

Visual Gesture BuilderRinse andrepeat

Your Application

Record example

gestures using Kinect Studio

Tag gestures using Gesture

Builder

Build and analyze

gestures using Gesture Builder

Preview gestures in

Gesture Builder

Call gesture detector for tracked bodies

Demo

Gesture Recognition

Kinect Fusion and Face

FusionProvides a rich mesh reconstruction of environment and objectsSurface reconstruction done using voxels

Fusion and Face SummaryFace

Offers a great entry-point into easy to use face attributesOperations and outputs are returned in 2D space

HD FaceProvides a high-definition mesh of a faceReconstruction is done by deforming “shape units”Operations and outputs are returned in 3D space! PAY TO PLAY !

Demo

Fusion (3D Reconstruction), and Face

Third Party Support

Third party supportIt’s easy to use Kinect APIs in your Unity3D app or gameKinect Common Bridge is the way to get to other frameworks (e.g. Cinder)Download KCB here -> http://aka.ms/kinectcommonbridgeCinder & openFrameworks -> http://aka.ms/k4wccsetup

Example: Motion Metrics

Experimental Implementation - VisualizeMotionMetricsIntegrate functionalities in controlling application

Visualize work per joint for linear and angular motion

Display velocity and acceleration vectors

Display energy expenditure rate and total energy expenditure

Display activity level

Display fall-detection state

Display detected posture

Center of mass

Enable/Disable “Jitter” filters

Enables logging

Demo

MotionMetrics

Evaluation: Energy Expenditure

Sensors of thearmband:

• Accelerometer• Heat flux• Near-Body Ambient temperature• Galvanic Skin Response Sensor• Skin Temperature

Perform physical exercise with 2 different models of a BodyMedia SenseWear armband worn at the same time

Test subject:• Male• 43 yrs• 178 cm• 72.6 kg• Non-smoker• right handed

Idea: Compare Kinect-based EE estimation with third-party EE estimation method

Evaluation: Validation of adjustments

-1 1 3 5 7 9 11 13 15 170

2

4

6

8

10

12

14

BodyMedia Kinect EE estimation comparison after corrective factors

Armband1 Armband2 KinectLin KinectAng KinectAvg

Runin place Pause

IntenseSquatsAndJumps

Lighter butmixed intensityrunningjumps/squats

Warmuprunningin place

Cooldown

Talk about Kinect for your organizationGet Sensor/Adapter: http://aka.ms/k4wv2purchaseDownload the SDK: http://aka.ms/k4wv2sdkTry samples and demos in the SDKTry apps and demos in Windows StoreBe creative, Brainstorm, Organize a hackathonCheck for announcements, events: http://kinectforwindows.com

Call to action

Questions & Contact

Andreas Erben

+1 (206) 349 2367+49 (179) 6944479v-anerbe@microsoft.comAndreas.Erben@daenet.comHackAndi.com

WIN-B210 Kinect with People and the Real WorldWednesday, October 29 5:00 PM - 6:15 PM Room: Hall 8.1 Room H

WIN-B308 The New Cocktail: 1 Tablet + 1 PC + 1 Phone + 1 Kinect + 1 Wall, Served Up on a CloudFriday, October 31 10:15 AM - 11:30 AM Room: Hall 8.1 Room K

WIN-B309 Kinect in Your Apps - Build to Amaze! Friday, October 31 8:30 AM - 9:45 AM Room: Hall 8.0 Room F4

Find me later: Andreas.Erben@daenet.com

Related content

General Info & Blog -> http://kinectforwindows.com

Track resources

Purchase Sensor -> http://aka.ms/k4wv2purchasev2 Preview SDK -> http://aka.ms/k4wv2sdk

Developer Forums -> http://aka.ms/k4wv2forum

Speaker’s blog -> http://hackandi.com

Windows 10http://aka.ms/trywin10

Stop by the Windows Booth to sign up for the Windows Insider Program to get a FREE Windows 10 T-shirt, whiles supplies last!

Windows Springboardwindows.com/itpro

Windows Enterprisewindows.com/enterprise

Windows ResourcesMicrosoft Desktop Optimization Package (MDOP)microsoft.com/mdop

Desktop Virtualization (DV)microsoft.com/dv

Windows To Gomicrosoft.com/windows/wtg

Internet Explorer TechNet http://technet.microsoft.com/ie

Resources

Learning

Microsoft Certification & Training Resources

www.microsoft.com/learning

Developer Network

http://developer.microsoft.com

TechNet

Resources for IT Professionals

http://microsoft.com/technet

Sessions on Demand

http://channel9.msdn.com/Events/TechEd

Please Complete An Evaluation FormYour input is important!TechEd Schedule Builder CommNet station or PC

TechEd Mobile appPhone or Tablet

QR code

Evaluate this session

© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

BACKUP

Fusion

2Create Reconstruction• Pass in a set of parameters defining the environment

and processing type• Outputs the reconstruction

3Other considerations• Can enable color to be lifted from the scan• Experimental pose finders (6DoF trackers) are also

included to play with

1Process Function• Provides “six degrees of freedom” transform• Can be rendered and use to begin the Fusion

Fusion | Face | HD Face

Face

Detection

Alignment

Orientation

Expressions

• Outputs a bounding box around the face

• Can be visualized in color or IR

• Identifies 5 facial landmarks on the face

• Operation performed in color or IR

• Returns quaternion of the head joint with respect to the sensor

• Quaternion prevents gimbal lock

• Provides classifiers for happy, left / right eye open, engagement, mouth open and mouth moving

Alignment and detection performs the operation in IR, but are converted to color using the coordinate mapper for output

Fusion | Face | HD Face

HD Face

Face Model Builder• Once of cluster of

classes that implement the capture interaction

• Interactive API

• Provides collections status and evaluation of frames you’ve collected

1

Face Model• Set of 94 shape units,

scale, hair color, and skin color

• All adjustments are set against an “average face” mesh that is deformed by the shape units acquired during the face capture

2

Use the Mesh• The mesh is analogous

to most animation meshes for easy application to other rigs

• Mesh topology is the same for all faces represented in a standard way (numTriangles, numVertices, etc.)

3

UX is a non-trivial consideration in a good implementation of the Face Model Builder.

Fusion | Face | HD Face

Fall detection

“Violent” fallInterval

to fall down

•Sum of max downwards force per joint over threshold

•Important joint (spine, shoulders, hips) close to floor

Interval to

recover

•Important joint locations above threshold height

“Slow” FallInterval to fall down

•Head position moves downward more than threshold value•Base of spine and head close to floor

Interval to recover

•Head and spine not in recovery threshold distance

Physical formulas

Body model, parameters

(WPF) rendering engine

(.NET)EventHandlers

Kinect for Windows SDK

APIs

Velocity, Acceleration,

Energy...

Activity & Fall detect.

Rule evaluation, fall detection state

machine

Activity Level,Fall detection

rules,...

K4W Runtime

Kinect V2 Camera

Kinect Drivers

KinectService

Posture recognition

Posture classification

Posture templates or

rules