DEVELOP YOUR MIXED REALITY APP FOR HOLOLENS WITH UNITY AND VISUAL STUDIO

ALEXANDER MEIJERSARCHITECT / HOLOLENS EVANGELIST @ ETTU

“HoloLens Evangelist”

I am responsible for architecture related business for our project services

department at ETTU. I’m involved as architect at different companies like Jumbo

Maritime, PGGM, Gemeente Amersfoort and others

Alexander Meijers

ABOUT ME Solutions Architect for ETTU

Founder of the Mixed Reality User Group

MixUG: http://www.mixug.nl

Email: a.meijers@ettu.nl or alexander@appzinside.com

Twitter: @ameijers

Blog: http://www.appzinside.com

3INTRO ETTU

TODAY’S TALKAGENDA Realities explained

Microsoft HoloLens

Building blocks and tools

Tools

Gestures

Sound

Spatial Mapping & Occlusion

Slider

Content Ads

MIXED REALITY

HOLOBASICS 5

REALITIESEXPLAINED

Merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interactin real time.

An overlay of synthetic content on the real world that is anchored to and interacts with the real world

MIXED REALITY

Direct or indirect view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data

An overlay of content on the real world where that content is not anchored to or part of it

AUGMENTED REALITY

Generation of realistic images, sounds and other sensations that replicate a real environment or create an imaginary setting

An immersive experience created entirely from computer-generatedContent. Also similar to 360 degree video

VIRTUAL REALITY

MICROSOFT HOLOLENS

HOLOBASICS 7

MICROSOFT HOLOLENSSPECIFICATIONS

Windows 10 device based on 32 bit architecture

Contains CPU, GPU and HPU

1GB Holographic Processor Unity (HPU)

64GB flash / 2GB memory

Device is more powerful than a laptop or game computer

No overheating due to warm air flows to the sides

2-3 hours active and 2 weeks standby

Weight 579g

Wi-fi

Gestures, Voice input and spatial understanding

HOLOBASICS 8

SENSORS, OPTICS ANDSPEAKERS

Environment camera

Depthcamera

Videocamera

Spatial SoundWith

speakers

High definitionlenses

HOLOBASICS 9

HOLOLENS EXAMPLES

“DEMONSTRATION OF SPATIAL DESIGN CONCEPT APP”

“HoloBasics by Alexander Meijers”

BUILDING BLOCKS AND TOOLS

HOLOBASICS 12

DEVELOPMENT TOOLS

Visual Studio 2017 UWP workload

Game development with Unity workload

Windows 10 SDK (version 1511 or later) There is no separate SDK for HoloLens

Unity 2017.2 or Unity 5.6.x Scripting Runtime Version Experimental .NET4.6. Equivalent

HoloLens emulator HoloLens Emulator (build 10.0.14393.1358)

Hyper-V

Contains DirectX project templates for Visual Studio

HoloLens device

GitHub Microsoft/MixedRealityToolkit-Unity

HOLOBASICS 13

BUILD LIFECYCLE OF AHOLOLENS PROJECT

Create Unity Project Configure HoloLens settings Create scene

Visual Studio

Unity HoloLens

Configure build settings Build and generate Visual Studio

project

Open project with Visual Studio Pair with HoloLens Build & deploy Visual Studio project

Start Application Test & debug Monitor

Unity

HOLOBASICS 14

WINDOWS DEVICE PORTAL

3D View

Mixed Reality Capture

Performance Performance tracing

System performance

Processes

Apps Maintenance

Crash dumps

Additional tools Logging

File Explorer

Virtual Input

HOLOBASICS 15

UNITYEDITOR

MainCamera

DirectionalLight

Game Objects

Assets

Components

Script

Scene

HOLOBASICS 16

UNITYPREFABS

• Predefined Game Objects

• Contains components with predefined values

• Prefabs used in Scene inherit the components and settings

HOLOBASICS 17

UNITYGAMEOBJECT SCRIPT

• Derived from MonoBehaviour

• Contains methods which are called by Unity through the a broadcast messaging system

• void Awake() – Called when class is instantiated

• void Start() – Called when object is enabled once before the first Update()

• void Update() – Called once per frame

• More methods are available• FixedUpdate, LateUpdate, OnGUI, OnDisable, OnEnabled

HOLOBASICS 18

UNITYMESSAGE SYSTEM

public void BroadcastMessage(string methodName, object parameter = null, SendMessageOptions options = SendMessageOptions.RequireReceiver);

• Used for calling methods on GameObjects

• Calls the methods also on its Children

• Examples• this.BroadcastMessage(“OnSelect”);

• focusedObject.BroadcastMessage(“OnSelect”);

HOLOBASICS 19

UNITYCOMPONENTS

• Transform• Position of object

• Rotation of object

• Scale (size) of object

• Mesh Renderer & Mesh Filter• Renders the object

• Mostly done using a mesh

• Mesh Collider• Determines the collision for the object

• Script• Perform actions on the object

• Handle messages on an object

“</CODE>”

“HoloBasics by Alexander Meijers”

HOLOBASICS 21

COORDINATE SYSTEMHOLOLENS AND UNITY

forward (x, y, z)Camera (0, 0, 0)Rotation (x, y, z)

Camera (1, -1, 4)Rotation (x, y, z)

HologramObject Pinned

HoloLensmoved to

new location

View directionGaze

GESTURES

HOLOBASICS 23

GESTURESGAZE

• The direction of the HoloLens pointing is called the gaze

• Hits an object or part of the environment

• Uses a Vector3 based value to define its so called “forward” direction

HOLOBASICS 24

GESTURESGESTURE RECOGNIZER

• Recognize input by tracking the position of either or both hands

• Gesture frame (Frustum)

• Recognized input from hands• Bloom

• Press, hold and release

• Gestures• Hold

• Manipulation

• Navigation

HOLOBASICS 25

GESTURESGESTURE RECOGNIZER

• Connect commands to gestures• GestureRecognizer gestures = new GestureRecognizer();

• gestures.TappedEvent += (source, tapCount, ray) => { … };

• gestures.StartCapturingGestures();

• Events• Tapped

• Select press and release

• Hold (Started, Completed, Canceled)

• Select press beyond system hold threshold

• Navigation (Started, Updated, Completed, Canceled)• Select press with relative movement

• Velocity based continuous scrolling or zooming

• Virtual Joystick

• Manipulation (Started, Updated, Completed, Canceled)

• Select press with absolute movement

• Move, resize or rotate hologram

HOLOBASICS 26

GESTURESSPEECH RECOGNIZER

• Use words or sentences to control your environment

• English language only at the moment

• Connect commands to spoken text• Dictionary<string, System.Action> keywords = new Dictionary<string,

System.Action>();

• keywords.Add(“some text”, () => { … };

• KeywordRecognizer keywords = new KeywordRecognizer(keywords.Keys.ToArray());

• keywordRecognizer.OnPhraseRecognized += OnPhraseRecognized;

• keywordRecognizer.Start();

• OnPhraseRecognized( PhaseRecognizedEventArgs args);• Invoke the action

• keywords[args.text].Invoke();

“</CODE>”

“HoloBasics by Alexander Meijers”

SPATIAL SOUND

HOLOBASICS 29

SPATIAL SOUNDOVERALL

• Simulates 3D sound using direction, distance and environmental simulations

• Above, below, behind, to the side, etc.

• Attach sound to holographic objects• Works also when object is not in line of

sight

• Used to draw attention

• Audio engine in HoloLens• CPU and memory considerations

• 10-12 spatial sound voices

HOLOBASICS 30

SPATIAL SOUNDTYPES

• Gaze mixing• Highlighting objects

• Play a sound on the object to get the users attention

• Audio Haptics• Reactive audio for touchless interactions

• Play a sound when user selects an object or when his hands appear inside the gesture frame

• Immersion• Ambient sounds surrounding the user

• Support your scene with background sounds or music

SPATIAL MAPPING & OCCLUSION

HOLOBASICS 32

SPATIAL PERCEPTIONHOW DOES IT WORK?

• Spatial perception and mapping

• Rooms

• Different ways of detecting and working with spatial perception

• Device scans your environment and builds a digital model in real time

• It allows HoloLens to see different surfaces, walls and ceiling

• Meshes & planes

• Possible to simulate a physical space

• E.g. Projecting a terrain over your floor

Spatial perception is the ability to be aware of

your relationships with the environment around

you and with yourself

“DEMONSTRATION OF LIVE SPATIAL MAPPING AND UNDERSTANDING BY HOLOLENS”

“HoloBasics by Alexander Meijers”

HOLOBASICS 34

SPATIAL MAPPING ANDUNDERSTANDING

Spatial Mapping components in Unity Spatial Mapping Renderer script

Spatial Mapping Collider script

Use an empty GameObject in your scene

Spatial Mapping render state None

Visualization

Occlusion

Rendering material Occlusion material

Visual material

Cursor on spatial mapping SpatialMappingCollider.layer = 31;

Cursor uses Raycast with layer mask 1<< 31

HOLOBASICS 35

OCCLUSIONCULLING

Process of determining which surfaces and parts are not visible from a certain viewpoint

Create a more naturally feeling with mixed reality

Holograms can be occluded by Other holograms

Real-life objects in your environment which are spatially mapped

Culling enforces the HoloLens not to (partially) render the object To speed up the rendering process

Objects which are too far away are left out the view (frustum)

Use Spatial Mapping Renderer Set Occlusion material

Set SpatialMappingRenderer.renderState = RenderState.Occlusion;

“</CODE>”“DEMONSTRATION OF SPATIAL MAPPING AND

OCCLUSION WITH HOLOGRAM FOR TRUE MIXED REALITY”

“HoloBasics by Alexander Meijers”

THANKS FOR WATCHING!

HoloBasics code on gitHub - https://github.com/ameijers/HoloBasics

Kampenringweg 45b, Gouda

0182-686 000

a.meijers@ettu.nl / alexander@appzinside.com

Contact me:

facebook.com/alexandermeijers.5

@ameijers

Follow me on: