Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012:...
Transcript of Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012:...
![Page 1: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/1.jpg)
1
Microso(KinectDepthSensing
![Page 2: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/2.jpg)
2
PeterSperlC#developeratAnylineStudentatTUWien(MedicalInformaCcs)GamedeveloperhobbyistEmail:[email protected]
![Page 3: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/3.jpg)
3
Agenda– Microso(Kinect
• About• Timeline• Components• SpecificaCons
– Depthsensing• V1
– Codedlight– TriangulaCon
• V2– TimeofFlight
– LiveDemo
![Page 4: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/4.jpg)
4
• MoConsensinginputdevice• Naturaluserinterfacethroughvoiceandgestures• InnovaCveuse-cases
– Gaming,HealthCare,Fitness,Medicine,VR,AR,RoboCcs,etc.
• Sensor&SDK:ForWindowsPCandXbox–Codewri[eninC#,C++/CLIorVB.NET
![Page 5: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/5.jpg)
5
Timeline
– 1stgeneraCon:• 2010:ReleaseforXbox360• 2011:KinectforWindowsSDK1.x• 2012:KinectforWindowsV1
– 2ndgeneraCon:• 2013:ReleaseforXboxOne• 2014:KinectforWindowsV2+SDK2.0• 2015:
– KinectforWindowsV2productwasdisconCnued,BUT:– WindowsuserssimplyuseKinectforXboxOnesensor+Kinectadapter
![Page 6: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/6.jpg)
6
Components• Colorsensor:
• commoncamerasensortoacquireRGBimages
• Depthcamera:• IRemi[er+IRsensoruseinfraredlightto
acquiredepthimages• V1:StructuredLight/CodedLight• V2:TimeofFlight
• Audiosensor:• Microphonearraytocapturesoundsignal
andsoundposiCon
![Page 7: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/7.jpg)
7
KinectSpecificaCons
Feature V1 V2
Colorstream 640x480at30FPS 1920x1280at30FPS
Depthstream 320x240 512x424
Infraredstream --- 512x424
Depthdistance 0.4m(nearmode)–4m 0.4m–4.5m
Horizontalfieldofview 57° 70°
VerCcalfieldofview 43° 60°
#definedskeletonjoints 20 25
#bodiestracked 2(+4recognized) 6
OperaCngsystem Windows7/8 Windows8
![Page 8: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/8.jpg)
8
KinectSpecificaCons
V1 V2
![Page 9: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/9.jpg)
9
DepthSensing(V1) 1. IREmi[er(E)projectsaknowninfraredlightpa[ern
(“codedlight”)intothescene2. IRSensor(S)retrievesreflectedlightfromthe
environment3. DepthmapiscomputedusingtriangulaConandlight
codingtechnology(byPrimeSense)4. DepthdataavailableviaUSBtotheAPI
![Page 10: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/10.jpg)
10
CodedLight(alsocalledstructuredlight)
• Emi[erprojectspseudo-randompa[ernoflight(“Speckles”)
• Pa[ernis“coded”:– hastobeuniqueper
posiConsoeachpointcanberecognizedinthepa[ern
![Page 11: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/11.jpg)
11
CodedLight• Lightisdistorted
onthesurface• Pa[ernisunique
ineveryposiCononthescene
• AllowstocomputedepthinformaConthroughtriangulaCon
![Page 12: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/12.jpg)
12
TriangulaCon
• UsedinstereovisiontoreconstructapointPin3Dspace,giventwoormoreimages
• MulCpleimageprojecConsofthesamesceneyielddepthinformaCon
• PosiCons,orientaCons,andopCcalproperCesoftheimagesourcesmustbeknown
![Page 13: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/13.jpg)
13
TriangulaCon
• Problem:– TriangulaConrequirestwoormoreimagesources,butKinecthasonlyonedepthsensor
![Page 14: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/14.jpg)
14
TriangulaCon
• Trick:Kinectactuallyusestwoimages:– TheimagecapturedfromtheIRsensor– The“invisible”hardwiredreferencepa[ern
• TriangulaConapplicablebecausethereisadistancebetweenIRsensorandIRprojector
![Page 15: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/15.jpg)
15
TriangulaCon
• Simplifiedmodel:– CamerasareparallelalongbaselineB– Cameracentersareatthesameheight– Camerashavesamefocallengthf
• Goal:calculatedisparityforeachpixel
B
![Page 16: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/16.jpg)
16
TriangulaCon
D=x–x’=(B*f)/z
z=(B*f)/(x–x’)
• DisparityD:DifferenceinimagelocaConofthesamepointXwhenprojectedfromdifferentperspecCves
• DisparityisinverselyproporConaltodepth
f Focallength
D Measureddisparity
B Baselengthbetweenemi[erandcamera
z DistancetoX
![Page 17: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/17.jpg)
17
TriangulaCon
• IRPa[erniscorrelatedagainstthereferencepa[ern• Projectedspeckles:
– ifdistancetothesensorissmallerorlargerthanonthereferenceplane:shi(edindirecConofthebaselinebetweenprojectorandperspecCvecenteroftheIRcamera
– x-direcConshi(saremeasuredforeachspeckle
• Distancefromeachpixeltothesensorisretrievedfromcorrespondingdisparity
xk,yk Coordinatesofpointk
xo,yo Principalpointoffsets
σx,σy LensdistorConcorrecCons
Z0 DistancebetweenobjectandSensoronreferenceplane
Zk DistancebetweenpointKandSensorinobjectspace
![Page 18: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/18.jpg)
17
TriangulaCon
• IRSensorandIRCameraareepipolarrec6fiedØ BothhavematchingFieldofViewØ Correspondencesearchreducedto1DproblemØ IfreflecCngsurfacegetsfartheraway,lightfromprojectorgets“larger”sinceeach
dotisaconeoflightØ Lightexpandingrateandcamerapixelexpansionarethesame
• Whenobjectdistancechanges,dotsappeartobeshi(edonlyinx-direcCon
• Trackingadot’sx-coordinateyieldsthepixelsobserveddepth
![Page 19: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/19.jpg)
18
DepthSensing(V2)
• 2ndgeneraConofKinectusesTimeofFlighttomeasuredepth
• IndirectmeasurementofCmeittakesforlaserpulsestoasurfaceandbacktothesensor
Ø Howisthispossible?
![Page 20: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/20.jpg)
19
TimeofFlight • Speedoflight=299,792,458m/s• 1GHzCPU=1*109Hz=1,000,000,000Hz->Pulseoflighttravels~29cmwithin1CPUcycle
• IRSensor:• divides1pixelinhalf• Halfpixelisturnedonandoffextremelyfast
• On:Absorbphotonsoflaserlight• Off:Rejectphotonsoflaserlight
• Otherhalfofthepixeldoesthesame,but180°phaseshi(ed
![Page 21: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/21.jpg)
20
TimeofFlight • LaserIlluminator(atthesameCme):
• Pulselightsourceinphaseofthefirstpixel-half
• Only“on”whenfirstpixelhalfis“on”
• Distanceat0cm:• Lightwillbeabsorbedinthe1sthalf,
nothingwillhappeninthesecondhalf
• Let’smovelightsourceawayforabit:• TimeofFlightwillslightlyincrease,photons
willreturninthe2ndhalfandberejected• Totalamountoflightabsorbedwill
decreaseslightly,totalamountoflightrejectedwillincreaseslightly
![Page 22: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/22.jpg)
21
TimeofFlight • Movingsourceevenfartheraway:
• Absorbedamountdecreasesfurther,rejectedamountincreasesfurther
• A(erseveralmillisecondsexposure:• Twototalamountsofphotonsabsorbedbythetwofirsthalvesarecompared
tothetwototalamountsofphotonsrejectedbythetwosecondhalves• RaCoisimportant,notthetotalamount–surfacecanabsorblight,butit
affectsbothmeasurementsequally
• Depthismeasuredbytheround-tripdistance
![Page 23: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/23.jpg)
22
TimeofFlight • Movingevenfurtheraway:
• Photonsmightovershoot1stand2ndhalfandarriveinthenextcycle• ->Ambiguity
• Ambiguityeasilyresolvedbyincreasingpixel-halfCmes,butthensmalldepthchangescan’tbemeasured
• SoluCon:• OnelowresoluConmeasurementasesCmate• Onehighprecisionmeasurement,usingesCmatetoeliminateambiguiCes
![Page 24: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/24.jpg)
24
LiveDemo
![Page 25: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/25.jpg)
25
Thankyou!
AndthanksPRIPforlendingmetheKinectforXboxOnesensorJ
Pa[ernRecogniConImageProcessingGroup@TUWien
![Page 26: Microso( Kinect · • 2010: Release for Xbox 360 • 2011: Kinect for Windows SDK 1.x • 2012: Kinect for Windows V1 – nd2 generaon: • 2013: Release for Xbox One • 2014: Kinect](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb6a2d36c407004e37f8afd/html5/thumbnails/26.jpg)
• Sources:– h[ps://msdn.microso(.com/en-us/library/hh973078.aspx#Depth_Ranges– AccuracyandResoluConofKinectDepthDataforIndoorMappingApplicaCons
h[ps://courses.cs.washington.edu/courses/cse455/09wi/Lects/lect16.pdf– h[p://www.slideshare.net/KouroshKhoshelham/generaCon-and-weighCng-of-3d-point-correspondences-for-improved-registraCon-of-rgbd-data– h[ps://mirror2image.wordpress.com/2010/11/30/how-kinect-works-stereo-triangulaCon/– h[p://people.eng.unimelb.edu.au/kkhoshelham/research_files/kinectaccuracy/isprsarchives-XXXVIII-5-W12-133-2011.pdf– h[p://www.cs.cornell.edu/~hema/rgbd-workshop-2014/papers/Alhwarin.pdf– h[p://www.visionrt.com/content/core-technology
• Images:– h[p://www.qualitymag.com/arCcles/91010--d-imaging-enters-the-machine-vision-world– h[p://www.cnblogs.com/TracePlus/p/4138615.html– h[ps://msdn.microso(.com/en-us/library/jj131033.aspx– h[p://vision.ia.ac.cn/Students/gzp/kinect1.html– h[p://2.bp.blogspot.com/-f4tcmi3LrDE/UZ0uY5f2p6I/AAAAAAAAB0Y/8FQlnBG8suA/s1600/new-kinect-internal.png
• CodeExample:– h[p://kinect.github.io/tutorial/