Applying Virtual Reality, and Augmented Reality to the Lifecycle...
Transcript of Applying Virtual Reality, and Augmented Reality to the Lifecycle...
Applying Virtual Reality, and Augmented Reality to the Lifecycle
Phases of Complex Products
Rich RabbitzChris Crouch
Copyright © 2017 Lockheed Martin Corporation. All rights reserved..
For Official Use Only 2
Lockheed Martin Moorestown, New Jersey
■ Lockheed Martin Rotary and Mission Systems (RMS)
● Designs, manufactures, services and supports
Military and Civil Helicopters
Naval and Radar Systems
Provides World-Class Systems Integration Training and
Logistics
● Lockheed Martin Moorestown, New Jersey
Naval and Radar Systems
World-class Systems Integrator
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 3
Lockheed Martin Moorestown, New Jersey
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 4
Lockheed Martin Moorestown, New Jersey
■ Ship Integration & Test
● 3D Space Arrangements for Naval Ships
● Topside Design for Naval and Ground Based Systems
● Modernization of Naval Ships
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 5
Tenets:
Copyright © 2013-2015 by Lockheed Martin Corporation. All rights reserved.
Surface Navy Innovation Center (SNIC)
A research, development and demonstration lab dedicated to innovating
affordable solutions across the maritime domain.
Technical Domain Focus:
• Training
• Cybersecurity/IA
• Big Data Analytics
• Models and Simulation
• Additive Manufacturing
• Mobile/Remote Workforce
• Advanced HMI/GUI Displays
• Advanced Display Technologies
• Augmented / Virtual Reality
• Situational Awareness
• Lifecycle Engineering
• Architecture/Design
• Cloud Computing
• Mission Planning
Driving Innovation, Affordability, and Capability
Put the Warfighter First – SNIC enables innovative concept
exploration, rapid prototyping, and risk reduction activities for
current and emerging Naval capabilities.
Accelerate Capability to the Warfighter – SNIC accelerates the
push to adapt available technology and capabilities for maritime
use.
Offer Flexible, Modular, Adaptable Solutions – SNIC utilizes an
agile methodology and enabling architecture to respond quickly
to new requirements and technologies.
Set the Standard for Collaboration – SNIC establishes an open
community space for government, industry, and academia.
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 6
Surface Navy Innovation Center (SNIC)
Collaboration
Loft
Unclassified
Lab
Equipment
Room
Conference
Room
Classified
Lab
Copyright © 2017 Lockheed Martin Corporation.
Rack of 4
VCAs
24 M6000s
8 P6000
3 P6000
2 M6000
4 K6000
Offsite20 K6000s
20 M6000s
For Official Use Only 7
Lifecycle Phases of a Product
■ Engineering and Deployment of Complex Products
● Naval Combat Ship
● Land Based Facilities
■ Lifecycle Phases of Complex Products
● Conceptual Design
● Production
● Deployment
● Upgrades
■ For which phases of the Lifecycle process are Virtual Reality
(VR), Augmented Reality (AR), and Mixed Reality (MR)
applicable?
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 8
COMITS Software Suite
■ COMITS (OpenGL based)● (Computerized Object Manipulation In Three-dimensional Space)
■ COMITS-Ray
● Physics Based Renderer (PBR)
● Real-time Interaction
● Photorealistic 360 VR Panorama images
● High Quality Video Production
■ COMITS-Ray Cloud
● Photorealistic streaming on the company intranet
■ COMITS-VR (Virtual Reality)
● Real-time OpenGL VR App
● OpenGL 4.5 / DesignWorks® / VRWorks®
■ COMITS-AR (Augmented and Mixed Reality)
● Direct3D®
● NVIDIA® GPU rendering via Holographic remoting
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 9
Bidirectional Data Exchange with CATIA® and AutoCAD®
DWG
Translator
Visual Basic
Macros
CATIA
AutoCAD
COMITS
NavisWorks
Translator
Ship Constructor
Creo
.obj
Rhino
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 10
COMITS-Ray
COMITS-Ray
Iray SDK
• Accomplished in 2012 via the NVIDIA Iray® SDK
• GTC 2017 (S7351) Applying GPU Technology to Combat
System Integration and Maintenance
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 11
COMITS-Ray Cloud Using AI
■ Interactive Streaming in a Chrome® Web Browser
● F35 model
● A ship model
Copyright © 2017 Lockheed Martin Corporation.
9.5 million triangles
For Official Use Only 12
COMITS-Ray with AI
■ AI Denoiser: Enhance Rendering Quality, SIGGRAPH 2017 talk
by Martin-Karl Lefrançois
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 13
Lifecycle Phases
■ Conceptual Design
● Product fully represented is 3D Computer Aided Design
(CAD) system
● Bring 3D CAD data into a VR system
Can the inhabitants of the complex product operate
and maintain it?
○ Ergonomics
− Can personnel comfortably
− See what they need to?
− Reach what they need to?
− Navigate to where they need to?
○ Collision resolution (multiple CAD systems)
○ Many, many, many more use cases
● Measurements
● Allow multiple local or remote users to collaborate
● Allow for model manipulations with changes send back to
CAD system
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 14
COMITS-VR
■ Running on NVIDIA® P6000 with HTC Vive®
● 9.6 million triangles
● 189 – 213 Frames Per Second (FPS) in standalone mode
● 90 FPS sustained with Vive® attached
● Plans for further optimizations and visual enhancements
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 15
COMITS-VR
■ 360 Panoramas generated from COMITS-Ray
■ Viewable in Vive®, or Oculus®, or on smart phones
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 16
Lifecycle Phases
■ Production
● Augmented Reality
Assembly Line Guidance
Extend the information for the shop floor workers
Drawings and diagrams
Videos of assembly sequence
● Virtual Reality
Equipment Installation Planning
● Mixed Reality
Visual Cues mixed in with the real world
These visual cues help guide the operator through a
series of assembly or maintenance steps
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 17
Lifecycle Phases
■ Deployment
● Product Familiarization
● Operations Training
● Maintenance Training
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 18
Virtual Reality
■ Product Familiarization
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 19
Virtual Reality
■ Operations
● Virtual Sandbox
● Multi-User VR
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 20
Augmented Reality
Copyright © 2017 Lockheed Martin Corporation.
Maintenance
For Official Use Only 21
Product Modernization
■ Product Upgrades
● Need a representation for the “As-Built” model
● Two choices
Go to the ship or facility
○ Mixed Reality
Bring the ship or facility back
○ Laser Scans
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 22
COMITS-AR
■ Microsoft HoloLens®
● Mixed Reality
● Holographic display
● Voice recognition
● Spatial sound
● Gesture recognition
● Spatial mapping
● Untethered
● 2-3 hour battery life
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 23
Mixed Reality
■ Fitting holographic cabinet through real door
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 24
COMITS-AR
■ Microsoft HoloLens®
● Can display models up to ~80,000 polygons while running
on 60 FPS
● Larger models lag in the display
■ How do we overcome this limitation?
■ Image generation
● Constant image size per frame
● One image required for left and right eye per frame
● 720p or 1268x720 per eye
■ Use GPU to generate left and right eye images
● Send these two images to HoloLens® device via Wi-Fi
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 25
GPU-AR
■ Install the Microsoft® Remoting Player app
■ Establishes a connection with the HoloLens and server with
one or more GPUs
■ The connection is made using standard Wi-Fi
■ Receives a data stream with input from a HoloLens
■ Renders content in a virtual holographic view
■ Streams content frames back to the HoloLens in real-time
■ This two-way connection allows the HoloLens to send
sensory and control information to the server
● Gestures
● Voice commands
● Spatial mapping
■ Server processes this information, updates the state of the
application, and sends images and sound back to the
HoloLens® in a constant stream
■ 64 bit application vs 32 bit
● More accuracy
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 26
GPU-AR
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 27
COMITS (OpenGL Based)
■ Real-time Point-Cloud Rendering (utilizing P6000)
■ 120 million points with sustained rendering at 60 FPS
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 28
COMITS (OpenGL Based)
■ Real-time Point-Cloud Rendering (utilizing P6000)
■ 120 million points with sustained rendering at 60 FPS
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 29
COMITS (OpenGL Based)
■ Real-time Point-Cloud Rendering (utilizing P6000)
■ 120 million points with sustained rendering at 60 FPS
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 30
COMITS (OpenGL Based)
■ Real-time Point-Cloud Rendering (utilizing P6000)
■ A billion points rendered
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 31
COMITS (OpenGL Based)
■ Real-time Point-Cloud Rendering (utilizing P6000)
■ GPU memory paging
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 32
COMITS (OpenGL Based)
■ Future plans to view these point clouds in VR and AR
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 33
COMITS (OpenGL Based)
■ Future plans to view these point clouds in VR and AR
■ Overlay with CAD data
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 34
COMITS
■ We are truly leveraging NVIDIA® technology to the fullest!
■ Apply VR and AR To Lifecycle Phases
● Conceptual Design
● Production
● Deployment
● Upgrades
■ Apply VR and AR To
● Catch problems early in Conceptual Design
● Training
● Operations
● Maintenance
● Readiness
■ Evolving 3D representation of platform
■ Reduce Lifecycle cost
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 35
Acknowledgements
■ The Ship Integration and Test (SI&T) Organization at LM
■ Chris Etgen, Director of SNIC
■ Eric Halpern, Lead Developer of COMITS-Ray
■ Chris Crouch, Lead Developer of COMITS-VR & COMITS-AR
■ Joshua Pacana, Lead Developer of COMITS
■ Brady White, VR content creator
■ NVIDIA®
● Craig Fullman
● John Chaney
● Martin-Karl Lefrancois
● Stefan Radig
● Robert Menzel
● And many, many others…
■ My mentors
● Dr. Norm Badler, University of Pennsylvania
● Dr. John Weaver, West Chester University
Copyright © 2017 Lockheed Martin Corporation.
For Official Use Only 36
Links to references
■ (S7351) Applying GPU Technology to Combat System
Integration and Maintenance by Rich Rabbitz
● Slides
http://on-demand.gputechconf.com/gtc/2017/presentation/s7351-
richard-rabbitz-applying-gpu-technology-combat-system-
integration-maintenance.pdf
● Recording
http://on-demand.gputechconf.com/gtc/2017/video/s7351-
christopher-crouch-applying-gpu-technology-to-combat-system-
integration-and-maintenance.mp4
■ AI Denoiser: Enhance Rendering Quality
by Martin-Karl Lefrançois
http://on-demand.gputechconf.com/siggraph/2017/video/sig1753-
martin-karl-lefrancois-train-your-own-denoiser.html
Copyright © 2017 Lockheed Martin Corporation.
Copyright 2017 Lockheed Martin Corporation 37Copyright © 2017 Lockheed Martin Corporation.
Rich Rabbitz
Lockheed Martin RMS
Principal Member of Engineering Staff (PMES)
Ship Integration and TestEmail: [email protected]