Sébastien Kuntz!Sebastien.Kuntz@imin-vr.com!CEO

The Challenges of!Creating VR Applications!

in Unity

!

• Founder & President!

• 12 years in VR!

!

• - French railways!

!

•

Sébastien Kuntz

• Simplify VR!

• Creation!

• Deployment!

• So YOU can spend more time creating applications

Plan• Unity!• Challenges!

• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Rendering speed!

• MiddleVR for Unity!• Conclusion

Plan

Unity!• Challenges!

• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

• Platform for creating interactive 3d applications!

• Widely used!

• 2 million registered developers!

• Certified by the US Army and Air Force!

• Clients : Bigpoint, Cartoon Network, Coca-Cola, Disney, Electronic Arts, LEGO, Microsoft, NASA, Nickelodeon, Ubisoft, US Army, Warner Bros.

• High-quality rendering!

• Deferred renderer!

• Occlusion culling!

• Realtime shadows!

• Ambient occlusion!

• Light mapping

Unity 3D Features• Programming!

• C#, Javascript!

• Profiler!

• Physics!

• NVidia PhysX!

• Cloth!

• Soft/rigid bodies!

• Ragdolls!

• Cars

3D formats !

FBX, Collada, 3DS, OBJ…!

Audio engine!

Networking!

Terrain

Editor

• Simple to use!

• Create your own tools inside Unity

Asset store

• Buy / sell assets!

• 3D models!

• Textures!

• Shaders!

• Scripts!

• Editor plugins

Desktop

Plan• Unity!

Challenges!• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

Plan• Unity!• Challenges!

Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

Virtual reality

Presence in a virtual world

(c) eMagin(c) Barco

Experience of presenceis in your brain !

Cognitive presencePerceptive presence

• Build hardware

To create and maintain presence, you need to ...

• Build software

• Build user experience

Plan• Unity!• Challenges!

• Presence!

Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

Device management

Physical device!!

Driver Application

Device management

Tracker 1! Driver Tracker 1

Application

Tracker 2!!

Driver Tracker 2

Plan• Unity!• Challenges!

• Presence!• Device management!

Display management!• Cluster management!• Interactions!• Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

Correct perspective

(c) Johnny Lee

Correct perspective

Stereoscopy

Warping & Blending

Graphics cards limitations

• Number of outputs!

• Processing power

multiple graphics cards

• Pro!

• More outputs!

• More processing power!

• Con!

• 3D engines are not optimized for multi-pipe

Cluster

• Pro!

• More outputs!

• More processing power!

• Con!

• 3D engines don’t support clustering

• No seams across displays require 3 layers of sync!

• 3D objects synchro (Scenelock)!

• New images display (Swaplock)!

• Left/Right eyes (Genlock)

Cluster

Plan• Unity!• Challenges!

• Presence!• Device management!• Display management!• Cluster management!

Interactions!• Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

Interactions

• Many different 3D user interactions!

• Navigation!

• Selection!

• Manipulation!

• Draw!

• System control!

• etc.

Interactions are dependent on hardware

Menus

Haptics (force feedback)

Plan• Unity!• Challenges!

• Presence!• Device management!• Display management!• Cluster management!• Interactions!

Deployment!• Speed!

• MiddleVR for Unity!• Conclusion

Deployment

• Modify application to!

• Change drivers!

• Change displays!

• Change interactions

Plan• Unity!• Challenges!

• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!

Rendering speed!• MiddleVR for Unity!• Conclusion

Rendering speed

Plan• Unity!

Challenges!• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Rendering Speed!

• MiddleVR for Unity!• Conclusion

Application

Plan• Unity!• Challenges!

• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!

MiddleVR for Unity!• Conclusion

• Build hardware

To create and maintain presence, you, the author, need to ...

• Build software

• Build user experience

“VR is mainly a software problem”!Jaron Lanier

What is MiddleVR ?

• VR plugin !

• Functionalities ○Manage input devices

•3D trackers, keyboard, mouse, joystick !

○Manage display •Viewports, cameras, stereoscopy, cluster

Desktop

http://www.youtube.com/watch?v=EEKH-1EilXs

MiddleVR tutorial

Device management

Device management

Supported Devices• Via VRPN :

• A.R.T • Vicon • Optitrack • Wiimote !

• Polhemus Fastrak Tracker, 3Space Tracker, Patriot Trackers,

• Ascension Flock of Bird • Intersense IS600, IS900 • 3dTech HiBall-3000 Wide Area Tracker, • Advanced Realtime Tracking Gmbh DTrack

Client, • World Viz Precision Position Tracker PPT

1.2, • Natural Point Optitrack Rigid Body Toolkit, • Phasespace optical tracking OWL, • ...

• Native drivers • Oculus Rift • Leap Motion • Kinect (Microsoft SDK) • PNI SpacePoint

Fusion • GameTrak • Razer Hydra • Motion Analysis (beta)

First integration of Haption’s haptic devices

http://www.youtube.com/watch?v=lnoE_VclnOI

Stereoscopy

• Active stereoscopy (OpenGL Quad-Buffer)!

• Which Unity can’t natively handle!

• Passive stereoscopy

Cluster & Multi-pipe

• Scenelock!

• Swaplock!

• Genlock!

• Multi-pipe (soon)

Supported displays

• Virtually any HMD • VR-Wall, Powerwall • Workbench, Holobench • HoloStage • CAVEs • 3D TVs • zSpace • Igloo Vision • …

Deployment

• Abstractions!

• Devices!

• 3D Nodes (User)!

• Interactions (soon)

Device management

Driver Tracker 1

Application

Driver Tracker 2

Device abstraction

Virtual Device

Application

Driver Tracker 1

Driver Tracker 2

Device management

Driver

Virtual Device Application

Simulation

Replay

Virtual Device 1

Hand (3D Node) Application

Virtual Device 2

Inverse Kinematics

User abstraction

Interactions• MiddleVR includes high-level interactions!

• Navigation!

• Selection!

• Manipulation!

• … More to come!

• Future: Interactions abstractions!

• Dynamically change interactions based on hardware

Interactions abstraction?Grab-air

Navigation

Application

Point

Steering

Selection

Manipulation

Raycasting

Go-go

Hand

Homer

Hand

WIM

Menus

• Ability to create menus and GUIs in HTML5 (soon)

Speed• Unity!

• Many options for optimizations!

• Occlusion culling!

• Static batching!

• Lightmap baking!

• etc.!

• 32-bit editor only!

• But player is 32-bit & 64-bit !!

• Subdivide parts in editor!

• Compositing!

• Cloud

Compositing Stereoscopy

Compositing Image decomposition

VR in the Cloud

• Temporary increase in power!

• Combine with compositing

Perceptive illusions

• Playing with limitations of human perception!

• Sensitivity of senses!

• Selectivity of attention!

• Useful to lower cost/complexity of VR systems

Perceptive illusions

(c) Franck Steinicke - Wurzburg University

Redirected walking

Change blindness :

(c) Evan Suma - USC/ICT

Perceptive illusionsChange blindness

Conclusion!

!

Thank you ! Questions ?

!Sebastien.Kuntz@imin-vr.com