Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo...
Transcript of Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo...
![Page 1: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/1.jpg)
Cem Cebenoyan
Director of Engineering, NVIDIA
Rendering Faster and Better with VRWorks
![Page 2: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/2.jpg)
Talk Overview
VRWorks Features
Context Priority, Direct Mode, VR SLI
Multi-Res Shading, Lens Matched Shading, Single Pass Stereo
UnrealEngine 4 Integration
![Page 3: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/3.jpg)
How is VR rendering different?
![Page 4: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/4.jpg)
How is VR rendering different?
High framerate, low latency
High FPS, low latency
Stereo Rendering
Lens Distortion
![Page 5: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/5.jpg)
NVIDIA VRWorks COMPREHENSIVE SDK FOR VR DEVELOPERS
![Page 6: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/6.jpg)
Context Priority and Direct Mode
![Page 7: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/7.jpg)
Context priority for asynchronous time warp Reduces latency from head rotation
Frame (Left)
Frame (Right)
Head Tracking (t)
Warped Frame
Head Tracking (t+1)
![Page 8: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/8.jpg)
TRIANGLES COMMAND PUSHBUFFER
PIXELS
PREEMPT
Pascal Graphics Preemption Improves responsiveness
![Page 9: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/9.jpg)
GPU
Latency
Scanout
Flip
Flash backlight
Conservative Preemption
Request
TIMEWARP BUFFER
Timewarp with Maxwell
![Page 10: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/10.jpg)
GPU
Latency
Scanout
Flash backlight
TIMEWARP
Pascal Preemption
Request
Flip
Timewarp with Pascal
![Page 11: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/11.jpg)
Direct Mode Plug-and-play Compatibility Via Direct Interface with the VR Headset
Operating System
![Page 12: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/12.jpg)
VR SLI
![Page 13: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/13.jpg)
VR SLI
Two eyes...two GPUs!
![Page 14: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/14.jpg)
“Normal” SLI
GPUs render alternate frames
N
N+1
N N+1
N N+1
CPU
GPU 0
GPU 1
Display
Latency
![Page 15: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/15.jpg)
VR SLI
Each GPU renders one eye—lower latency
NL
N+1R
N N+1
NR
N+1L
N N+1 CPU
GPU 0
GPU 1
Display
Latency
![Page 16: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/16.jpg)
VR SLI
GPU affinity masking: full control
Shadow maps,
GPU physics,
etc.
Left eye rendering
Right eye
rendering
UINT SetGPUMask( [in] UINT GPUMask
);
![Page 17: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/17.jpg)
VR SLI
Broadcasting reduces CPU overhead
R
L
Render scene
once
![Page 18: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/18.jpg)
VR SLI
Per-GPU constant buffers, viewports, scissors
L R
Multi-GPU
API R
L
Engine
NvAPI_Status VSSetConstantBuffers( [in] ID3D11DeviceContext *pContext, [in] UINT GPUMask, [in] UINT StartSlot, [in] UINT NumBuffers,
);
void glLGPUNamedBufferSubDataNVX ( GLbitfield gpuMask, GLuint buffer, GLintptr offset, GLsizeiptr size, const GLvoid *data );
![Page 19: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/19.jpg)
VR SLI
Cross-GPU data transfer via PCI Express
NvAPI_Status CopySubresourceRegion( [in] ID3D11DeviceContext *pContext, [in] ID3D11Resource *pDstResource, [in] UINT DstSubresource, [in] UINT DstGPUIndex, [in] UINT DstX, [in] UINT DstY, [in] UINT DstZ, [in] ID3D11Resource *pSrcResource, [in] UINT SrcSubresource, [in] UINT SrcGPUIndex, [in] const D3D11_BOX *pSrcBox, [in, optional] UINT ExtendedFlags = 0 );
void glLGPUCopyImageSubDataNVX ( GLuint sourceGpu, GLbitfield destinationGpuMask, GLuint srcName, GLuint srcTarget, GLint srcLevel, GLint srcX, GLint srcY, GLint srcZ, GLuint dstName, GLuint dstTarget, GLint dstLevel, GLint dstX, GLint dstY, GLint dstZ, GLsizei width, GLsizei height, GLsizei depth );
![Page 20: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/20.jpg)
Multi-Resolution Shading
![Page 21: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/21.jpg)
VR headset optics
Distortion and counter-distortion
![Page 22: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/22.jpg)
VR headset optics
Distortion and counter-distortion
User’s view Image Displayed Optics
![Page 23: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/23.jpg)
Distorted rendering
Render normally, then resample
Rendered image Distorted image
![Page 24: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/24.jpg)
Distorted rendering
Over-rendering the outskirts
Rendered image Distorted image
![Page 25: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/25.jpg)
Multi-resolution shading
Subdivide the image, and shrink the outskirts
![Page 26: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/26.jpg)
Multi-resolution shading
Fast viewport broadcast on NVIDIA Maxwell GPUs
Viewport 1
Viewport 2
Viewport N
...
Geometry Pipeline
![Page 27: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/27.jpg)
Lens Matched Shading RENDERS TO A LENS CORRECTED SURFACE
![Page 28: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/28.jpg)
LMS Image Original Image
Lens Matched Shading RENDERS TO A LENS CORRECTED SURFACE
![Page 29: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/29.jpg)
TRADITIONAL STEREO RENDERING REQUIRES 2 GEOMETRY PASSES
Left Eye (Pass 1) Right Eye (Pass 2)
![Page 30: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/30.jpg)
Traditional Stereo Rendering
Double the full pipeline rendering cost
Many engines now adding support for Instanced Stereo Rendering
Improves on CPU side cost of stereo submission
Doesn’t help with GPU cost
![Page 31: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/31.jpg)
SINGLE PASS STEREO RENDERS LEFT & RIGHT EYE IN ONE GEOMETRY PASS
Left Eye
Right Eye
![Page 32: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/32.jpg)
Demo!
![Page 33: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/33.jpg)
Multi-Res in UnrealEngine 4
![Page 34: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/34.jpg)
Multi-resolution shading
We’ve integrated multi-res in UE 4.10, 4.11, and 4.12
Currently limited support for post effects with multi-res
Available on GitHub
https://github.com/NvPhysX/UnrealEngine/tree/MultiRes-4.10
https://github.com/NvPhysX/UnrealEngine/tree/MultiRes-4.11
https://github.com/NvPhysX/UnrealEngine/tree/MultiRes-4.12
Pascal features coming soon
Unreal Engine integration
![Page 35: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/35.jpg)
UE4 integration 4.10
Postprocessing passes supported:
Bloom & lens flare
Tonemapping
Temporal AA
Reflection environment (not screen-space reflection)
Height fog
Decals
Distortion/refraction
https://github.com/NvPhysX/UnrealEngine/tree/MultiRes-4.10
![Page 36: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/36.jpg)
UE4 integration 4.11 / 4.12
Adds postprocessing passes supported:
SSAO
Screenspace Reflections
Supports Instanced Stereo Rendering
Major refactoring
https://github.com/NvPhysX/UnrealEngine/tree/MultiRes-4.11, MultiRes-4.12
![Page 37: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/37.jpg)
Multi-resolution shading
UE4 Infiltrator demo: +30% to +40% FPS
@ approximate VR render res
Everest VR seeing up to +40%
Best when pixel-bound
We’ve seen ~50% perf boosts
Performance
![Page 38: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/38.jpg)
UE4 integration
Direct engine integration of Multi-res shading
RHI support for multiple viewports & scissors
FastGS plumbing
Post-processing shaders
Main points
![Page 39: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/39.jpg)
Rendering passes
Geometry passes: add FastGS; render with 9 viewports & scissors
Base pass, decals, deferred lights, depth prepass, distortion, shadow projection, translucency, velocity
Image-space passes: remap linear/multi-res UVs as needed
Decals, deferred lights, postprocessing, shadow projection, tiled deferred lights
![Page 40: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/40.jpg)
Image-space passes
Reconstruct 3D position from 2D+depth
Map 2D position to linear before reconstructing
Large filter kernels
Map sample point to linear, apply offset, map back to multi-res
Two common cases
![Page 41: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/41.jpg)
UV remapping
![Page 42: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/42.jpg)
Shadow projection
Before:
Example of reconstructing 3D position
![Page 43: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/43.jpg)
Shadow projection
After:
Example of reconstructing 3D position
![Page 44: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/44.jpg)
Shadow projection
After:
Example of reconstructing 3D position
![Page 45: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/45.jpg)
Shadow projection
After:
Example of reconstructing 3D position
![Page 46: Rendering Faster and Better with VRWorksMulti-Res Shading, Lens Matched Shading, Single Pass Stereo UnrealEngine 4 Integration How is VR rendering different? How is VR rendering different?](https://reader030.fdocuments.net/reader030/viewer/2022040617/5f218b6a2f349c67b517041f/html5/thumbnails/46.jpg)
Demo Time
Demo UE4!