Code europe holoadvanced - building more advanced mixed reality apps for hololens
42
BUILDING MORE ADVANCED MIXED REALITY APPS FOR HOLOLENS HOLOADVANCED
-
Upload
alexander-meijers -
Category
Technology
-
view
66 -
download
0
Transcript of Code europe holoadvanced - building more advanced mixed reality apps for hololens
BUILDING MORE ADVANCED MIXED
REALITY APPS FOR HOLOLENS
HOLOADVANCED
“HoloLens Evangelist”
I am responsible for architecture related business for our project services
department at ETTU. I’m involved as architect at different companies like Jumbo
Maritime, PGGM, Gemeente Amersfoort and others
Alexander Meijers
ABOUT ME Solutions Architect for ETTU
Founder of the Mixed Reality User Group
MixUG: http://www.mixug.nl
Email: [email protected] or [email protected]
Twitter: @ameijers
Blog: http://www.appzinside.com
4INTRO ETTU
TODAY’S TALKAGENDA Realities Explained
HoloLens
Mixed Reality Toolkit
Spatial perception
Coordinate system & anchoring
Advanced gestures
Performance & stabilization
Slider
Content Ads
MIXED REALITY
HOLOBASICS 6
REALITIESEXPLAINED
Merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interactin real time.
An overlay of synthetic content on the real world that is anchored to and interacts with the real world
MIXED REALITY
Direct or indirect view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data
An overlay of content on the real world where that content is not anchored to or part of it
AUGMENTED REALITY
Generation of realistic images, sounds and other sensations that replicate a real environment or create an imaginary setting
An immersive experience created entirely from computer-generatedContent. Also similar to 360 degree video
VIRTUAL REALITY
MICROSOFT HOLOLENS
HOLOBASICS 8
MICROSOFT HOLOLENSSPECIFICATIONS
Windows 10 device based on 32 bit architecture
Contains CPU, GPU and HPU
1GB Holographic Processor Unity (HPU)
64GB flash / 2GB memory
Device is more powerful than a laptop or game computer
No overheating due to warm air flows to the sides
2-3 hours active and 2 weeks standby
Weight 579g
Wi-fi
Gestures, Voice input and spatial understanding
HOLOBASICS 9
SENSORS, OPTICS ANDSPEAKERS
Environment camera
Depthcamera
Videocamera
Spatial SoundWith
speakers
High definitionlenses
HOLOBASICS 10
HOLOLENS EXAMPLES
HOLOBASICS 12
MIXED REALITY TOOLKIT FOR UNITYCOMMON
Collection of scripts and components intended to accelerate development of applications
Maintained on GitHub by Microsoft and contributors
Target Microsoft HoloLens
Windows Mixed Reality headsets
Integrated in Unity with package
Tools Windows 10 FCU
Unity 3D
Visual Studio 2017
HOLOBASICS 13
MIXED REALITY TOOLKIT FOR UNITYTARGETS
“WALK THROUGH OF HOLOLENS APPLICATIONUSING TOOLKIT”
“HoloAdvanced by Alexander Meijers”
SPATIAL PERCEPTION
HOLOBASICS 16
SPATIAL PERCEPTIONHOW DOES IT WORK?
• Spatial perception and mapping
• Rooms
• Different ways of detecting and working with spatial perception
• Device scans your environment and builds a digital model in real time
• It allows HoloLens to see different surfaces, walls and ceiling
• Meshes & planes
• Possible to simulate a physical space
• E.g. Projecting a terrain over your floor
Spatial perception is the ability to be aware of
your relationships with the environment around
you and with yourself
“DEMONSTRATION OF DEFAULTSPATIAL MAPPING”
“HoloAdvanced by Alexander Meijers”
HOLOBASICS 18
SPATIAL PERCEPTIONSPATIAL UNDERSTANDING
HoloLens is always scanning the world around us
Understanding your environment by distinguish different surfaces
Converting environment raw meshes to logical flat planes
Map planes to different surfaceslike ceiling, floor, wall, floor-like and other
HOLOBASICS 19
SPATIAL PERCEPTIONDIFFERENT METHODS
Method How Part of More information
Standard components Use the standard components SpatialMappingRenderer and SpatialMappingCollider to map the environment
Unity Microsoft
Shaders By converting a vertex location to world space and checking the vertex normal if a pixel is vertical
Unity Academy
SurfaceMeshesToPlanes This class helps you identify wall, floor, ceiling and table by mapping meshes to actual planes and is written in C#
Mixed Reality Toolkit
(opensource)
Academy
SpatialUnderstanding Getting a higher level of understanding of the world around you by analyzing the environment using a engine build in C++. Developed by Asobo Studios with Microsoft for games like Fragments and Young Conker
Mixed Reality Toolkit
(opensource)
Case study
Custom Use SurfaceObserver classes from the Spatial Mapping API to build your own custom spatial understanding system
Spatial Mapping API Microsoft
“DEMONSTRATION OF SPATIAL UNDERSTANDING OF THE ENVIRONMENT”
“HoloAdvanced by Alexander Meijers”
HOLOBASICS 21
SPATIAL PERCEPTIONROOM SCANNING
One-time Best for (almost) static environments
Holograms don’t have the need to react to the environment
Continuous Best for dynamic environments
Holograms need to react to the environment
COORDINATE SYSTEM & ANCHORING
HOLOBASICS 23
COORDINATE SYSTEMHOLOLENS
forward (x, y, z)Camera (0, 0, 0)Rotation (x, y, z)
Camera (1, -1, 4)Rotation (x, y, z)
HologramObject Pinned
HoloLensmoved to
new location
View directionGaze
HOLOBASICS 24
ANCHORINGFRAME OF REFERENCES VERSUS ANCHORS
• Moves with the user as they walk around
• Content is often called body-locked content
• Standing-scale or room-scale experiences within a 5 meter diameter
• Suitable for static holograms moving with the user
ATTACHED FRAME OF
REFERENCE
• In principle Unity’s world coordinates
• Content is often called world-locked content
• Standing-scale or room-scale experiences within a 5 meter diameter
• Suitable for (highly) dynamic/moving holograms
STATIONARY FRAME
OF REFERENCE
• Represents an important point in the world that should be keep tracked of
• Has its own coordinates
• Experiences which are beyond a 5 meter diameter
• Suitable for holograms that remain fixed in the world
• Suitable for holograms that move slowly/smoothly within the anchor’s area
SPATIAL ANCHORS
HOLOBASICS 25
ANCHORINGSPATIAL ANCHORS
Represents an important point in the world that is tracked by the system in time
Has its own coordinate system
Adjust when needed relative to other anchors or frame of reference
Achieves the most accurate position for a hologram
Very cheap to create
myHologramObject.AddComponent<WorldAnchor>();
HOLOBASICS 26
ANCHORINGWHEN TO USE SPATIAL ANCHORS
In most cases create a spatial anchor for each hologram placed in the scene by an end user
Keep the following in mind
You can use a single spatial anchor for multiple holograms
Render holograms always within 3 meters of their attached anchor
Avoid creating a grid of spatial anchors to allow holograms move between different anchored areas
Clean-up spatial anchors which you not use anymore
HOLOBASICS 27
MICROSOFT HOLOLENSSHARING ACROSS SESSIONS
Persist holograms across instances of the application
WorldAnchorStore class
Important methods
WorldAnchorStore.GetAsync(OnStoreLoaded)
WorldAnchorStore.Save(“some unique id”, myAnchor);
WorldAnchorStore.Load(“some unique id”, myGameObject);
WorldAnchorStore.GetAllIds();
HOLOBASICS 28
MICROSOFT HOLOLENSANCHOR SHARING BETWEEN DEVICES
Persist holograms across HoloLens devices
Collectively view and interact with the same holograms
Each device must have scanned enough of the environment to position the anchor
WorldAnchorTransferBatch class
Important methods
WorldAnchorTransferBatch.ExportAsync(transferBatch, onDataAvailable, onCompleted);
WorldAnchorTransferBatch.ImportAsync(serializeData, offset, length, onComplete);
“DEMONSTRATION OF MAINTAINING ANCHORS ACROSS SESSIONS”
“HoloAdvanced by Alexander Meijers”
ADVANCED GESTURES
HOLOBASICS 31
ADVANCED GESTURESGESTURE RECOGNIZER
• Recognize input by tracking the position of either or both hands
• The combination of hand gestures and gaze allows us to create a target system
• Two core gestures• Air tap
• Bloom
• Gesture frame • Nose to waist
• Shoulder to Shoulder
• Gesture recognizer allows you to configure which gestures are targeted on
• Composite gestures
HOLOBASICS 32
ADVANCED GESTURESCOMPOSITES
By combining tap, hold, release and movement of the hand allows us to create more complex gestures
Also called composite gesture or high level gesture
Be aware that long complex composite gestures could break when the user relaxes their hand
Composite gestures
Manipulation Can be used for move, resize or rotate hologram with a 1:1 hand reaction
Paint with your hands
Navigation Operate like a virtual joystick
Can be used for navigating in UI experiences
“DEMONSTRATION OF A COMPOSITE GESTURE”
“HoloAdvanced by Alexander Meijers”
HOLOBASICS 34
ADVANCED GESTURESINTERACTION
HoloLens is also able to track the users hand position and velocity
Tracking will only start when the user has a “ready” position of the hand Air tap
Bloom
InteractionManager events InteractionSourceDetected
InteractionSourceUpdated
InteractionSourceLost
35INTRO ETTU
“DEMONSTRATION OF (SAFE) FIREUSING HOLOLENS”
“HoloAdvanced by Alexander Meijers”
OPTIMALIZATION & STABILITY
HOLOBASICS 38
HOLOLENS PERFORMANCEOPTIMALISATIONS
Especially important when building more complex HoloLens applications
Keep your FPS as close to as 60 FPS
<35FPS will increase instability of your HoloLens App
Maintain frame-rate consistency
Watch out for compatibility between Unity and Visual Studio versions!
HOLOBASICS 39
HOLOLENS PERFORMANCEINSTABILITIES
Type What you wil see How to resolve
Jitter Shaking of holograms tracking environment degrades. Use sensor tuning
Judder uneven motion and doubling of holograms maintain constant 60 FPS
Drift holograms move away from their original point Use spatial anchors and make sure the environment environment is mapped
Jumpiness holograms jumps around Make sure the environment is mapped or not changed much
Swim holograms move smoothly corresponding to to the movement of the HoloLens
Calibrate for the user and/or check if hologram is is part of stabilization plane
Color separation Rainbow effect around the edges of the hologram
hologram moves to quickly and/or try to maintain a a constant 60 FPS
HOLOBASICS 40
HOLOLENS PERFORMANCETHINGS YOU CAN DO
Spatial mapping only when needed and if possible low number of triangles
Don’t use Mixed Reality Capture via Windows Device Portal
Prevent rendering of holograms outside the frustum
Remove spatial anchors you do need. The system will keep prioritize sensor data around these anchors
Remove directional light if not needed
HOLOBASICS 41
HOLOLENS PERFORMANCETHINGS YOU CAN DO
Adjust stereo rendering (but be aware!)
Enable Low Latent Frame Presentation (but be aware!)
Use low poly of hologram objects
Use HoloLens specific shaders
Use of “static” on GameObjects which do not move
Keep code as clean as possible. Remove unused assets
HOLOBASICS 42
WRAP-UPTAKAWAYS
Mixed Reality Toolkit delivers a number of prefabs and components which allows you to build quicker
There are different methods for spatial mapping and understanding. Use what's best for you
Spatial Anchors allow you to share across sessions and even devices
Build more complex composite gestures by using standard gestures, hand movement and location
Keep optimizing your application to maintain stability. Keep FPS at 60
THANKS FOR WATCHING!
HoloAdvanced code on gitHub - https://github.com/ameijers/HoloAdvanced
Kampenringweg 45b, Gouda
0182-686 000
[email protected] / [email protected]
Contact me:
facebook.com/alexandermeijers.5
@ameijers
Follow me on:
