Pyramid Algorithm Framework for Real-Time Image Effects in Game Engines

Pyramid Algorithm Framework for Real-Time ImageEffects in Game Engines

Tuesday 3rd May, 2016

Adrià Arbués Sangüesa - Andreea-Daniela Ene - Nicolai KroghJørgensen -Christian Aagaard Larsen - Daniel Michelsanti - Martin

Kraus

24

Pyramid AlgorithmFramework

Adrià ArbuésSangüesa et al.

IntroductionBlur

Convolutional Methods

Pyramid Methods

Motivation

FrameworkGame Engine

Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

Appendices

Agenda

IntroductionBlurConvolutional Methods

Pyramid MethodsMotivationFramework

Game EngineStructureEffectsBlurBloomDepth of Field

ResultsConclusionsReferencesAppendices

24



Introduction2 Blur


Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

Appendices

IntroductionBlur

24



IntroductionBlur

3 Convolutional Methods

Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

Appendices

IntroductionConvolutional Methods

∗ =

k ✕ k

Gaussian FilterN ✕ N

Input Image

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

Appendices


∗ =

k ✕ k

Gaussian Filter

2N ✕ 2N

Input Image

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

Appendices


I In order to have the same effect: double the size of thekernel.

I Complexity: O(N2 × k2)

24



IntroductionBlur


6 Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

Appendices

Pyramid Methods

How to reduce the complexity?I Pyramid:

I Linear complexity on the number of pixels of the inputimage [5].

24



IntroductionBlur


7 Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

Appendices

Pyramid Methods

Analysis Synthesis

24



IntroductionBlur


8 Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

Appendices

Pyramid Methods

Analysis Synthesis

24



IntroductionBlur


Pyramid Methods

9 Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

Appendices

Motivation

I No built-in support to use pyramid methods in many gameengines.

I Goal: build a framework to make pyramid methods easilyaccessible.

24



IntroductionBlur


Pyramid Methods

Motivation

Framework10 Game Engine

Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

Appendices


Global game engine market share:

Unity Closest competitor Others

Figure: [4]

24



IntroductionBlur


Pyramid Methods

Motivation


11 Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

Appendices

FrameworkStructure

Frame_1

Analysis List

Synthesis Dictionary

Key_1 Frame_1

Key_2 Frame_2

… …

Master Dictionary

Key_1 Synthesis_1

… …

Synthesis Dictionary

Key_n Synthesis_n

1024 x 1024

512 x 512

256 x 256

…

32 x 32

2048 x 2048

1024 x 1024

Generatenon power of tworesolution texture

1080 x 1920

512 x 512

Generatesynthesis

listSynthesis List

1080 x 1920

Screen

Generateanalysis list

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

12 Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

Appendices

FrameworkEffects

Figure: [1], [2], [3]

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

13 Blur

Bloom

Depth of Field

Results

Conclusions

References

Appendices

FrameworkBlur

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

Blur

14 Bloom

Depth of Field

Results

Conclusions

References

Appendices

FrameworkBloom

A

BC

A: Bright colors extractionB: Pyramidal blurC: Combination of the textures

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

15 Depth of Field

Results

Conclusions

References

Appendices

FrameworkDepth of Field

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

16 Depth of Field

Results

Conclusions

References

Appendices

FrameworkDepth of Field

FD

A A2 FS

FC

P

1024 x 1024

512 x 512

256 x 256

…

32 x 32

2048 x 2048 2048 x 2048

1024 x 1024 1024 x 1024

2048 x 2048

512 x 512

1080 x 1920

Analysis Synthesis 1 Synthesis 2 Synthesis 3Screen

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

17 Results

Conclusions

References

Appendices

Results

Figure: Performance of the Framework (ms)

Figure: VRAM Usage (MB)

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

18 Conclusions

References

Appendices

Conclusions

I Pyramid algorithm framework implemented in Unity.I Demonstrated and tested by implementing image effects.I Performed slightly worse than the built-in effects.I Expected results as the framework is a work in progress.

Thanks for your attention!

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

19 References

Appendices

References

[1] Blur effects.http://docs.worldviz.com/vizard/postprocess_blur.htm.Accessed: 01-03-2016.

[2] Simple bloom node setup.http://www.blendswap.com/blends/view/21849.Accessed: 01-03-2016.

[3] Understanding depth of field.http://www.24frameslater.com/2013/03/05/understanding-depth-of-field-episode-1.Accessed: 01-03-2016.

[4] Unreleased McKinsey report.

[5] Peter J Burt.Fast filter transform for image processing.Computer graphics and image processing, 16(1):20–51, 1981.

[6] Ondrej Fialka and Martin Cadik.Fft and convolution performance in image filtering on gpu.In Information Visualization, 2006. IV 2006. Tenth International Conference on, pages 609–614.IEEE, 2006.

http://docs.worldviz.com/vizard/postprocess_blur.htm

http://www.blendswap.com/blends/view/21849

http://www.24frameslater.com/2013/03/05/understanding-depth-of-field-episode-1

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

20 Appendices

AppendicesConvolutional Methods

How to reduce the complexity?I Separable kernel:

∗ =∗

I Complexity: O(N2 × 2k).I Still dependent on the kernel size.

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

21 Appendices

AppendicesConvolutional Methods

How to reduce the complexity?I Frequency domain:

. =FF-1

F

I Complexity: O(4N2 × log2(N)).I Filter-size independent, but not linear on the number of

pixels of the input image.I Suitable for non-separable big filters.

Further details on these approaches: [6].

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

22 Appendices

AppendicesFilters

2x2 Box

Analysis Filter4x4 Box

Analysis Filter

Biquadratic B-Spline

Analysis Filter

Biquadratic B-Spline

Synthesis Filter

Original ImageBlurred Image

2x2 Box Filter

Blurred Image

4x4 Box Filter

Blurred Image

Biquadratic B-Spline Filter

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

23 Appendices

AppendicesGraphics and Compute Pipelines

Compute Shader

Unordered Access

Resources

Shader Resources and

Constant Buffers

Tessellation

Index BuffersVertex Buffers

Input Assembler

Vertex Shader

Hull Shader

Tessellator

Domain Shader

Geometry Shader

Rasterizer

Pixel Shader

Output Merger

Render Targets

Unordered Access

Resources

Shader Resources and

Constant Buffers

Tessellation

Buffers

24



IntroductionBlur


Pyramid Methods

Motivation


Structure

Effects

Blur

Bloom

Depth of Field

Results

Conclusions

References

24 Appendices

AppendicesBlur

Pyramid Algorithm Framework for Real-Time Image Effects in Game Engines

Engineering

Transcript of Pyramid Algorithm Framework for Real-Time Image Effects in Game Engines