NUX Presentation from TechMixer Birmingham 2011
-
Upload
michael-heydt -
Category
Technology
-
view
52 -
download
2
Transcript of NUX Presentation from TechMixer Birmingham 2011
![Page 1: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/1.jpg)
NUX in a
nutshell
![Page 2: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/2.jpg)
![Page 3: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/3.jpg)
agenda• Introduction• What is NUI/NUX• NUX In the media and the wild• Real-world devices• Tools, SDK’s, …• Where is this going?• Demo using Kinect and Wiimote
![Page 4: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/4.jpg)
about mike• Principal global technologist, Advanced Technology Services, SunGard Global Services
• Technology focus on .NET, cloud, parallel systems, rich interfaces and NUX
• [email protected], [email protected]• @mikeheydt• 42spikes.com
![Page 5: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/5.jpg)
55
Advanced TechnologyWhat is Advanced Technology?
SunGard Consulting Services Approach
Typical Client Engagements
Our consultants offer deep domain experience and a broad set of leading edge technology skills, allowing companies to optimize their business processes
We use a variety of Agile project management and execution techniques to facilitate quick delivery and reduce risk
We take an objective approach in the selection of tools and products, and are comfortable working with a variety of vendor products and platforms
We have applied our advanced technology skills to provide solutions to customers in finance, energy and media sectors
Typical assignments involve deploying highly skilled specialists with relevant domain knowledge who work with customers in a collaborative way to design and
build custom solutions
MapReduce/HadoopComplex Events ProcessingGPU Mathematical Pricing
Mashups
Web 2.0Cloud & Grid ComputingFunctional LanguagesMobile Applications
RIA (Silverlight/Flex)Semantic Web
REST/JSONGrails
![Page 6: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/6.jpg)
![Page 7: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/7.jpg)
opening questions…
• Who here has used a Kinect on the Xbox?• Who has programmed a Kinect?• Who has used a WiiMote on the Wii?• Who has programmed a wiimote in their application?
• Who has seen Minority Report?
![Page 8: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/8.jpg)
what is nui/x?
• In computing, a natural user interface, or NUI, is the common parlance used by designers and developers of computer interfaces to refer to a user interface that is effectively invisible, or becomes invisible with successive learned interactions, to its users.
• NUI Covers the class of interfaces where the user is enabled to interact with the computer in a more natural manner.
![Page 9: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/9.jpg)
genesis• Conceptually has been thought of for several decades
• Made popular by Minority Report• iPad has changed the game with single-task gestural apps
• Xbox Kinect has broken open the flood gates for NUX
• NUX is starting to show adoption in operating systems through the use of tablet and trackpad gestures
![Page 10: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/10.jpg)
in the media and the wild
• Xbox• Operating systems• Movies and television• Science
![Page 11: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/11.jpg)
XBox
![Page 12: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/12.jpg)
os and mobility: osx lion
![Page 13: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/13.jpg)
os and mobility: Windows 8
![Page 14: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/14.jpg)
tv: Caprica
![Page 15: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/15.jpg)
movie: District 9
![Page 16: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/16.jpg)
movie: Disclosure
![Page 17: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/17.jpg)
movie: Minority Report
![Page 18: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/18.jpg)
movie: Minority Report
![Page 19: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/19.jpg)
movie: Minority Report
![Page 20: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/20.jpg)
science: world wide telescope
![Page 21: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/21.jpg)
other: smart cubes
![Page 22: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/22.jpg)
real world devices
• Surface• Kinect• PrimeSense• Wiimote• Tablets and SmartPhones
![Page 23: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/23.jpg)
surface
![Page 24: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/24.jpg)
surface capabilities
• Multi-user• Multi-touch• Also camera based identification and tracking• Built in wireless to devices
![Page 25: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/25.jpg)
surface 2.0
![Page 26: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/26.jpg)
kinect• Laser Depth Imager• Video Camera• Microphone Array• Very XBox Specific Interface (non-standard USB)• Microsoft SDK, non-commercial license• $150
![Page 27: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/27.jpg)
kinect: an evolution
![Page 28: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/28.jpg)
primesense• Only laser depth finder• Standard USB• OpenNI drivers• Semi-open source• commercial license
![Page 29: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/29.jpg)
johnny 5
![Page 30: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/30.jpg)
wiimote
![Page 31: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/31.jpg)
tablets and smartphones• Have driven
adoption of NUX• Will be an
integral “augmentation” to NUX
![Page 32: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/32.jpg)
elements of nux
• Skeleton Tracking• Gesture (Body, fingers) Identification• Voice Recognition
![Page 33: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/33.jpg)
the how of the kinect
• Perform initialization gesture (helps, but not always necessary)
• You are scanned and tracked by an IR laser• Depths are measured to the millimeter at 320 x 200 +
• From depths skeletons are identified and tracked
![Page 34: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/34.jpg)
Kinect SDK
http://research.microsoft.com/en-us/um/
redmond/projects/
kinectsdk/
![Page 35: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/35.jpg)
OpenNI• http://www.o
penni.org/
• Pseudo-open source
![Page 36: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/36.jpg)
data from the kinect
• Multiple streams of data• Video• Audio• Depth• Skeletons
![Page 37: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/37.jpg)
kinect - the invisible
![Page 38: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/38.jpg)
depth mapping• Distance to the millimeter• Used to create maps for “skeletons”
![Page 39: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/39.jpg)
skeleton tracking
• Built from the depth map• 1..n skeletons• Done in software not the hardware
![Page 40: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/40.jpg)
the “psi” gesture
![Page 41: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/41.jpg)
hand and finger tracking
Not a part of any SDK
![Page 42: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/42.jpg)
the how of wiimote
• Connects to PC with Bluetooth• With .NET, use WiimoteLib (http://wiimotelib.codeplex.com/)
![Page 43: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/43.jpg)
Wiimotelib
http://www.ted.com/talks/johnny_lee_demos_wii_remote_hacks.html
![Page 44: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/44.jpg)
Thoughts on the Wiimote• An interesting device, but…
• It’s first gen• Looked good prior to the Kinect• All capabilities can be duplicated with Kinect (point tracking) and a cell phone (accelerometer, gyro)
• And a cell phone can provide additional uses such as robust data entry
![Page 45: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/45.jpg)
speech• Kinect has a very good set of microphones• Speech recognition is not in the kinect - use the .NET libraries
• Must be bridged with Microsoft Speech Platform SDK
• I can’t stress this enough – this is excellent augmentation
![Page 46: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/46.jpg)
gesture recognition• Not built into any of the devices
• Typically implemented with Dynamic Time Warping algorithms (DTW)
• Record sequences of skeleton locations, match against live streams
![Page 47: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/47.jpg)
difficulties• Many devices• Immature libraries• Small body of work• Many API’s, even for each individual device
• Kinect is non-standard USB
• All devices are separate and don’t interoperate
• All work on a single system – No support for moving physical location
• No standard pattern language for NUX
• No “canned” gesture engines
![Page 48: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/48.jpg)
what i’m building• NuiDotNet (nuidotnet.codeplex.com, nuidot.net)
• Abstracts devices from the application• Allows configuration and location based construction of services
• Multiple and prioritized services and interactions mapped to application functions
• Gesture engine and domain specific language for gestures
![Page 49: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/49.jpg)
directions• Define useful set of gesture patterns for data manipulation
• Build finger / hand tracking capabilities• Make the DTW gesture engine robust• Tracking of specific parts of the body• Architecture and software to plug into business applications
• Provide a library of pluggable UI elements and gestures
![Page 50: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/50.jpg)
uses for business
• Interactive data manipulation and visualization• Free-form visual application development• Media presentations• Home control• Many still to be determined
![Page 51: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/51.jpg)
demo and the tools
• Hardware: Kinect, WiiMote• Libraries: Kinect SDK, Wiimotelib• Language and GUI: C# / WPF
![Page 52: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/52.jpg)
resources• ^ Buxton, Bill (2010-01-06). "CES 2010: NUI with Bill Buxton". Microsoft Research.
• http://www.microsoft.com/surface/en/us/default.aspx
• http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/
• http://www.youtube.com/watch?v=1-tMp4WkQjA
• http://openni.org/
• http://silverlightkinect.codeplex.com/
• http://mono-mig.sourceforge.net/
• https://42spikes.com
![Page 53: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/53.jpg)
Resources
• http://en.wikipedia.org/wiki/NUI• http://kinectdtw.codeplex.com/• http://en.wikipedia.org/wiki/Dynamic_time_warping
• http://tinyurl.com/6a64sqr
• http://wiimotelib.codeplex.com/
![Page 54: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/54.jpg)
Resources
• http://ctmarionette.wordpress.com/ (marionette)• http://www.ted.com/talks/johnny_lee_demos_wii_remote_hacks.html
• http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/docs/speech_walkthrough.pdf
• http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=14373
![Page 55: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/55.jpg)
resources
• http://www.springer.com/cda/content/document/cda_downloaddocument/9783540740476-c1.pdf?SGWID=0-0-45-452103-p173751818
• http://www.youtube.com/watch?v=tlLschoMhuE&feature=player_embedded
• http://www.youtube.com/watch?v=lCuItHQEgEQ&NR=1
![Page 56: NUX Presentation from TechMixer Birmingham 2011](https://reader031.fdocuments.net/reader031/viewer/2022022201/588acf471a28ab89058b5a2b/html5/thumbnails/56.jpg)
resources
• http://www.microsoft.com/presspass/press/2011/jan11/01-06mssurfacesamsungpr.mspx
• http://blogs.oracle.com/mprove/entry/reipadsnui• http://channel9.msdn.com/Shows/HanselminutesOn9/Hanselminutes-on-9-Johnny-Lee-on-Computer-Vision