DEVELOP YOUR MIXED REALITY APP FOR HOLOLENS

WITH UNITY AND VISUAL STUDIO

HOLOBASICS

“HoloLens Evangelist”

At ETTU I am responsible architecture related business for our project services

department at ETTU. I’m involved as architect at different companies like Planon,

PGGM, Gemeente Amersfoort and others

Alexander Meijers

ABOUT ME Solutions Architect for ETTU

Founder of the Mixed Reality User Group

MixUG: http://www.mixug.nl

Email: a.meijers@ettu.nl or alexander@appzinside.com

Twitter: @ameijers

Blog: http://www.appzinside.com

3INTRO ETTU

TODAY’S TALKAGENDA Mixed Reality

Microsoft HoloLens

Tools

Our first hologram

Sound

Gestures

Spatial Mapping

Occlusion

Slider

Content Ads

MIXED REALITY

HOLOBASICS 5

VIRTUAL REALITY

Generation of realistic images, sounds and other sensations that replicate a real environment or create an imaginary setting

An immersive experience created entirely from computer-generated Content. Also similar to 360 degree video

HOLOBASICS 6

AUGMENTED REALITY

Direct or indirect view of a physical, real-worldenvironment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data

An overlay of content on the real world where that content is not anchored to or part of it

HOLOBASICS 7

MIXED REALITY

Merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time.

Overlay of synthetic content on the real world that is anchored to and interacts with the real world

MICROSOFT HOLOLENS

HOLOBASICS 9

HISTORY OF HOLOLENS

Codenamed Project HoloLens

Chief inventor Alex Kipman Official title is technical fellow

He dreamed up Kinect in at the end of 2007 and set a vision which incorporated HoloLens

It is the start of a transforming world

In the new reality, sensors will be anywhere

A visual computing platform controlled by speech and gesture

Satya Nadella - “The emergence of the next computing interface”

Alex Kipman

HOLOBASICS 10

MICROSOFT HOLOLENSSPECIFICATIONS

Windows 10 device based on 32 bit architecture

64GB flash

2GB memory

1GB Holographic Processor Unity (HPU)

First of its kind

Device is more powerfull than a laptop

No overheating due to warm air flows to the sides

2-3 hours active and 2 weeks standby

Weight 579g

HOLOBASICS 11

SENSORS, OPTICS ANDSPEAKERS

Environment camera

Depthcamera

Videocamera

Spatial SoundWith

speakers

High definitionlenses

HOLOBASICS 12

MICROSOFT HOLOLENS

First Holographic computer running on Windows 10

Contains a CPU, GPU and HPU

Human understanding

Gesture and voice input

Gaze tracking

Spatial sound

Uses wi-fi to connect and has no wires attached

Design for comfort

HOLOBASICS 13

MICROSOFT HOLOLENSSPECIFICATIONS

Contains depth camera Field of vision that spans 120 by 120 degrees

18 sensors flooding the device with terabytes of data every second

Tricks your brain into perceiving holographic images Light engine in which light particles are bouncing million times

Photons enter the two lenses

Ricochet some between layers of blue, green and red glass

Hitting back of the eye

HOLOBASICS 14

HOLOLENS EXAMPLES

HOLOBASICS 15

HOLOLENS EXAMPLES

HOLOBASICS 16

HOLOLENS EXAMPLES

HOLOBASICS 17

HOLOLENS EXAMPLES

HOLOBASICS 18

HOLOLENS EXAMPLES

“DEMONSTRATION OF HOLOLENSAPPLICATIONS”

“HoloBasics by Alexander Meijers”

TOOLS

HOLOBASICS 21

DEVELOPMENT TOOLS

Visual Studio 2017 UWP workload

Game development with Unity workload

Windows 10 SDK (version 1511 or later) There is no separate SDK for HoloLens

Unity 5.6 or Unity 2017.1 Be aware! - Unity 2017.2 beta for immersive headsets

HoloLens emulator HoloLens Emulator (build 10.0.14393.1358)

Hyper-V

Contains DirectX project templates for Visual Studio

HoloLens device

HOLOBASICS 22

DEVELOPMENT TOOLSGITHUB

Microsoft/MixedRealityToolkit-Unity

Ameijers/HoloLens-Unity-Utilities

HOLOBASICS 23

BUILD LIFECYCLE OF AHOLOLENS PROJECT

Create Unity Project Configure HoloLens settings Create scene

Visual Studio

Unity HoloLens

Configure build settings Build and generate Visual Studio

project

Open project with Visual Studio Pair with HoloLens Build & deploy Visual Studio project

Start Application Test & debug Monitor

Unity

HOLOBASICS 24

DEVELOPMENT FORHOLOLENS

HoloLens Device Developer mode needs to be turned on

Pair the device with your development machine

Retrieve IP address of HoloLens Device Ask Cortana

Settings Network & Internet Advanced options

HOLOBASICS 25

WINDOWS DEVICE PORTAL

3D View

Mixed Reality Capture

Performance Performance tracing

System performance

Processes

Apps Maintenance

Crash dumps

Additional tools Logging

File Explorer

Virtual Input

HOLOBASICS 26

UNITYEDITOR

MainCamera

DirectionalLight

Game Objects

Assets

Components

Script

Scene

HOLOBASICS 27

UNITYPREFABS

• Predefined Game Objects

• Contains components with predefined values

• Prefabs used in Scene inherit the components and settings

HOLOBASICS 28

UNITYGAMEOBJECT SCRIPT

• Derived from MonoBehaviour

• Contains methods which are called by Unity through the a broadcast messaging system

• void Awake() – Called when class is instantiated

• void Start() – Called when object is enabled once before the first Update()

• void Update() – Called once per frame

• More methods are available• FixedUpdate, LateUpdate, OnGUI, OnDisable, OnEnabled

HOLOBASICS 29

UNITYMESSAGE SYSTEM

public void BroadcastMessage(string methodName, object parameter = null, SendMessageOptions options = SendMessageOptions.RequireReceiver);

• Used for calling methods on GameObjects

• Calls the methods also on its Children

• Examples

• this.BroadcastMessage(“OnSelect”);

• focusedObject.BroadcastMessage(“OnSelect”);

HOLOBASICS 30

UNITYCOMPONENTS

• Transform• Position of object

• Rotation of object

• Scale (size) of object

• Mesh Renderer & Mesh Filter• Renders the object

• Mostly done using a mesh

• Mesh Collider• Determines the collision for the object

• Script• Perform actions on the object

• Handle messages on an object

HOLOBASICS 31

TOOLS3D MODELING

Blender

Studio 3DS

Max

Many others like

Maya, Rhino3D

Paint 3D

OUR FIRST HOLOGRAM

HOLOBASICS 33

COORDINATE SYSTEMHOLOLENS AND UNITY

forward (x, y, z)Camera (0, 0, 0)Rotation (x, y, z)

Camera (1, -1, 4)Rotation (x, y, z)

HologramObject Pinned

HoloLensmoved to

new location

View directionGaze

HOLOBASICS 34

HOLOGRAMS AND PINNING

Device can project a hologram into a room and keep it locked in position

Called pinnnig

Objects are not moving relative to you but you can move around the object

Allows you to view the object from any angle

“DEMONSTRATION OF PINNED HOLOGRAM”

“HoloBasics by Alexander Meijers”

HOLOBASICS 36

PINNED HOLOGRAMSUMMARY

Basics of Unity and 3D objects

Building an app using prefabs, components and script

Build lifecycle of a HoloLens application using Unity and Visual Studio

Pinned hologram robot in the room continue facing the HoloLens wearer

SPATIAL SOUND

HOLOBASICS 38

SPATIAL SOUNDOVERALL

• Simulates 3D sound using direction, distance and environmental simulations

• Above, below, behind, to the side, etc.

• Attach sound to holographic objects• Works also when object is not in line of

sight

• Used to draw attention

• Audio engine in HoloLens• CPU and memory considerations

• 10-12 spatial sound voices

HOLOBASICS 39

SPATIAL SOUNDTYPES

• Gaze mixing• Highlighting objects

• Play a sound on the object to get the users attention

• Audio Haptics• Reactive audio for touchless interactions

• Play a sound when user selects an object or when his hands appear inside the gesture frame

• Immersion• Ambient sounds surrounding the user

• Support your scene with background sounds or music

HOLOBASICS 40

SPEECHCORTANA

• Start with “Hey Cortana”

• Some thing you can say• Move <app name> here

• Take a picture

• Start or stop recording

• Call <contact>

• How much battery do i have left?

• Restart or Shutdown

• Cortana can be turned on/off

“DEMONSTRATION OF SPATIAL SOUND ADDED TO HOLOGRAM”

“HoloBasics by Alexander Meijers”

HOLOBASICS 42

PINNED HOLOGRAMSPATIAL SOUND

Unity and audio sources

Creating a spatial sound with an audio source

Explained how the speakers of HoloLens adjust based on the location of the sound

GESTURES

HOLOBASICS 44

GESTURESGAZE

• The direction of the HoloLens pointing is called the gaze

• Hits an object or part of the environment

• Uses a Vector3 based value to define its so called “forward” direction

HOLOBASICS 45

GESTURESGESTURE RECOGNIZER

• Recognize input by tracking the position of either or both hands

• Gesture frame (Frustum)

• Recognized input from hands• Bloom

• Press, hold and release

• Following your hand

• Gestures• Hold

• Manipulation

• Navigation

HOLOBASICS 46

GESTURESGESTURE RECOGNIZER

• Connect commands to gestures• GestureRecognizer gestures = new GestureRecognizer();

• gestures.TappedEvent += (source, tapCount, ray) => { … };

• gestures.StartCapturingGestures();

• Events• Tapped

• Select press and release

• Hold (Started, Completed, Canceled)

• Select press beyond system hold threshold

• Navigation (Started, Updated, Completed, Canceled)• Select press with relative movement

• Velocity based continuous scrolling or zooming

• Virtual Joystick

• Manipulation (Started, Updated, Completed, Canceled)

• Select press with absolute movement

• Move, resize or rotate hologram

HOLOBASICS 47

GESTURESSPEECH RECOGNIZER

• Use words or sentences to control your environment

• English language only at the moment

• Connect commands to spoken text• Dictionary<string, System.Action> keywords = new Dictionary<string,

System.Action>();

• keywords.Add(“some text”, () => { … };

• KeywordRecognizer keywords = new KeywordRecognizer(keywords.Keys.ToArray());

• keywordRecognizer.OnPhraseRecognized += OnPhraseRecognized;

• keywordRecognizer.Start();

• OnPhraseRecognized( PhaseRecognizedEventArgs args);• Invoke the action

• keywords[args.text].Invoke();

“DEMONSTRATION OF GESTURES BASED ON HAND AND SPEECH TO INTERACT WITH

HOLOGRAM”

“HoloBasics by Alexander Meijers”

HOLOBASICS 49

PINNED HOLOGRAMGESTURES WITH HAND AND SPEECH

Implemented code for handling the gaze and cursor

Implemented code for adding gestures

Implemented code for adding speech

Control the robot by shooting at it with a click gesture

Control the scene by using words and sentences like “reset”

SPATIAL MAPPING

HOLOBASICS 51

SPATIAL PERCEPTIONHOW DOES IT WORK?

• Spatial perception and mapping

• Rooms

• Different ways of detecting and working with spatial perception

• Device scans your environment and builds a digital model in real time

• It allows HoloLens to see different surfaces, walls and ceiling

• Meshes & planes

• Possible to simulate a physical space• E.g. Projecting a terrain over your floor

Spatial perception is the ability to be aware of

your relationships with the environment around

you and with yourself

HOLOBASICS 52

SPATIAL MAPPING ANDUNDERSTANDING

Device scans your environment and builds a digital model in real time

It allows HoloLens to see different surfaces, walls and ceiling

Possible to simulate a physical space E.g. Projecting a terrain over your floor

“DEMONSTRATION OF LIVE SPATIAL MAPPING AND UNDERSTANDING BY HOLOLENS”

“HoloBasics by Alexander Meijers”

HOLOBASICS 54

PINNED HOLOGRAMSPATIAL MAPPING UNDERSTANDING

Windows Device Portal connected to the HoloLens device

Mapping the room by walking around

Show mapping in the Windows Device Portal

HOLOBASICS 55

SPATIAL MAPPING ANDUNDERSTANDING

Spatial Mapping components in Unity Spatial Mapping Renderer script

Spatial Mapping Collider script

Use an empty GameObject in your scene

Spatial Mapping render state None

Visualization

Occlusion

Cursor on spatial mapping SpatialMappingCollider.layer = 31;

Cursor uses Raycast with layer mask 1<< 31

Rendering material Occlusion material

Visual material

“DEMONSTRATION OF SPATIAL MAPPING USED WITH HOLOGRAM”

“HoloBasics by Alexander Meijers”

HOLOBASICS 57

PINNED HOLOGRAMSPATIAL MAPPING

Using the component SpatialMappingRenderer and SpatialMappingCollider in Unity

Show spatial mapping by using the HoloDeck material/shader

SPATIAL MAPPING & OCCLUSION

HOLOBASICS 59

OCCLUSIONCULLING

Process of determining which surfaces and parts are not visible from a certain viewpoint

Create a more naturally feeling with mixed reality

Holograms can be occluded by Other holograms

Real-life objects in your environment which are spatially mapped

Culling enforces the HoloLens not to (partially) render the object To speed up the rendering process

Objects which are too far away are left out the view (frustum)

Use Spatial Mapping Renderer Set Occlusion material

Set SpatialMappingRenderer.renderState = RenderState.Occlusion;

“DEMONSTRATION OF SPATIAL MAPPING AND OCCLUSION WITH HOLOGRAM FOR TRUE

MIXED REALITY”

“HoloBasics by Alexander Meijers”

HOLOBASICS 61

PINNED HOLOGRAMSPATIAL MAPPING AND OCCLUSION

Using the component SpatialMappingRenderer and SpatialMappingCollider in Unity

Change the way spatial mapping is done for None, Visualization and Occlusion

Placed another robot behind a real-life object and explain occlusion

HOLOBASICS 62

WRAP-UPTAKAWAYS

Unity and Visual Studio are the tools when it comes to developing mixed reality apps for HoloLens

Know your tools like the Windows Device Portal

Understand physics when creating an app for HoloLens

Lot’s of stuff to make your life easy for gestures, speech, spatial mapping and other

Understand the basics and how it works

Lot of stuff available on GitHub like the MixedReality Toolkit to make your life easier

THANKS FOR WATCHING!

HoloBasics code on gitHub - https://github.com/ameijers/HoloBasics

Kampenringweg 45b, Gouda

0182-686 000

a.meijers@ettu.nl / alexander@appzinside.com

Contact me:

facebook.com/alexandermeijers.5

@ameijers

Follow me on: