Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you...

59
Introduction to PowerVR Ray Tracing Tuesday 18th March, 2014 @ GDC James A. McCombe

Transcript of Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you...

Page 1: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Introduction to PowerVR Ray TracingTuesday 18th March, 2014 @ GDC

James A. McCombe

Page 2: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

What are we launching today?

Page 3: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

System Memory Bus

VertexData Master

PixelData Master

ComputeData Master

Course Grain

Scheduler Core Management

Memory Cache Unit

2D Core (PTLA)

Pixel Co-Processor

Tiling Co-processor

Control and Register BusHost CPU Interface

RayData Master

System Memory Interface

Frame Accumulator

Cache

Scene Hierarchy Generator

Intersection Processor Array Coherency Engine

Ray Tracing Unit

USC0 USC1Texture Unit

USCn-1 USCnTexture Unit

Unified Shading Cluster Array

...

Page 4: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive
Page 5: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive
Page 6: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

What are the applications for this?

Page 7: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive
Page 8: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Augmented Reality

Image and Diagram CopyrightPeter Kán 2012

Page 9: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Virtual Reality a.k.a Multi-persective rendering

Lenticular Displays

Ultra-low latency rendering

Lens distortion and aberration correction

Page 10: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Hybrid rendering in games

• Shadows

• Reflections

• Transparency

• Better scaling with multiple dynamic lights

• Real-time light map updates for the rasteriser

• Easy to bolt on

Page 11: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Fully ray traced graphics

• Brute force path tracing

• Produces photo realism easily

• Pretty much requires all the 3D content to be ray traced

• Possible in todays technology in console / desktop using Wizard

• Probably not practical for fully real-time use in mobile for a couple more generations. Fine for sub-realtime use.

Page 12: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Non-graphics

• Better in-game AI

• Collision Detection

• 3D spatial search

Page 13: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Overview of PowerVR

Page 14: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Power and Bandwidth

0W

75W

150W

225W

300W

Workstation Mobile Workstation Laptop Light Laptop Tablet0Gb/s

30Gb/s60Gb/s90Gb/s

120Gb/s150Gb/s180Gb/s210Gb/s240Gb/s270Gb/s300Gb/s

Graphics Watts Graphics Bandwidth

Page 15: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Capability of our mobile graphics

0

37.5

75

112.5

150

2006 2007 2008 2009 2010 2011 2012 2013

Shad

er G

Flop

s

• Almost 5x compute capability

• 4x vec4 MAD → 16x Scalar 2xMAD

• 1.5 → 3.6 Gtexels/s

• OpenGLES 3.0 Support

Page 16: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

System Memory Bus

VertexData Master

PixelData Master

ComputeData Master

Course Grain

Scheduler Core Management

Memory Cache Unit

2D Core (PTLA)

Pixel Co-Processor

Tiling Co-processor

Control and Register BusHost CPU Interface System

Memory Interface

USC0 USC1Texture Unit

USCn-1 USCnTexture Unit

Unified Shading Cluster Array

...

Page 17: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

• An instance is a vertex, pixel or OpenCL thread

• All instances in the task share Program, Uniforms, Parameters, etc.

Compute Task

32 Instances

Vertex Task

32 Instances

CGS

USC0 USC1Texture Unit

USCn-1 USCnTexture Unit

Unified Shading Cluster Array

Pixel Task

32 Instances

VertexData

Master

PixelData

Master

ComputeData

Master

Page 18: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

ALU

Common Store Registers (Shared across instances in the task)

ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU

Unified Store Registers (Private registers per instance)

CommonInstruction

Per Pixel / Vertex data and shader temps

Shader Uniforms and Constants

(not exact)

Page 19: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Output 1

Output 0

Source 0

Source 1

Source 2

Source 3

Source 4

Source 5

ABS, NEG, etc.

ABS, NEG, etc.

ABS, NEG, etc.

ABS, NEG, etc.

ABS, NEG, etc.

ABS, NEG, etc.

⩽ 2 Flops

Integer Math

Format Conversions

⩽ 2 Flops

Comparison Format Conversions

12 Cycles of Latency (not exact)

Page 20: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

ALU

Common Store Registers (Shared across instances in the task)

ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU ALU

Unified Store Registers (Private registers per instance)

CommonInstruction

varying vec2 texCoord;

void main() { if (texCoord.x > 0.5) { ... ... } else { ... ... }

Page 21: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Overall capability

• Series6 gives you over 100Gflops for shading on latest mobile devices

• OpenGL ES 3.0 support

• Massive capability gain allowing math heavy shaders

• Mobile graphics is where the actual innovation is happening

• Try it!

Page 22: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

A Technical introduction to Wizard

Page 23: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

System Memory Bus

VertexData Master

PixelData Master

ComputeData Master

Course Grain

Scheduler Core Management

Memory Cache Unit

2D Core (PTLA)

Pixel Co-Processor

Tiling Co-processor

Control and Register BusHost CPU Interface

RayData Master

System Memory Interface

Frame Accumulator

Cache

Scene Hierarchy Generator

Intersection Processor Array Coherency Engine

Ray Tracing Unit

USC0 USC1Texture Unit

USCn-1 USCnTexture Unit

Unified Shading Cluster Array

...

Page 24: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

System Memory Bus

VertexData Master

PixelData Master

ComputeData Master

Course Grain

Scheduler Core Management

Memory Cache Unit

2D Core (PTLA)

Pixel Co-Processor

Tiling Co-processor

Control and Register BusHost CPU Interface

RayData Master

System Memory Interface

Frame Accumulator

Cache

Scene Hierarchy Generator

Intersection Processor Array Coherency Engine

Ray Tracing Unit

USC0 USC1Texture Unit

USCn-1 USCnTexture Unit

Unified Shading Cluster Array

...

Page 25: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Ray RAM

Ray RAM

Output 1

Output 0

Source 0

Source 1

Source 2

Source 3

Source 4

Source 5

ABS, NEG, etc.

ABS, NEG, etc.

ABS, NEG, etc.

ABS, NEG, etc.

ABS, NEG, etc.

ABS, NEG, etc.

⩽ 2 Flops

Integer Math

Format Conversions

⩽ 2 Flops

Comparison Format Conversions

12 Cycles of Latency (not exact)

Page 26: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

New shader type

• Ray Shaders are invoked when a ray intersects a triangle

• Shaders types can emit any number of rays

• OpenRL has a frame shader to emit primary rays

• Existing fragment / pixel shaders can emit rays also!

• GLSL Programming model is the same

• Built-in functions, parameters, etc.

Page 27: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Primitive Objects

• Encapsulate the rendering state for one mesh

• Includes VBOs, uniforms, shader programs, texture bindings, etc.

• Ray tracing unit sorts rays into tasks with common primitive objects

• Persistent between frames

• Mutable objects by the client

Page 28: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Constraints / Limitations

• Shaders cannot wait for results of individual ray trace operations

• Shaders must provide a worst case estimate on number of child rays

• Per-ray user data payload must be carefully managed

Page 29: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Parallelism is on rays NOT pixelsTask 0

Task 3

Task 2

Task 1

ALU ALU ALU ALUCommonInstruction

(In this example, ALU width is 4 to make the diagram smaller)

Page 30: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Parallelism is on rays NOT pixels

Page 31: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Relief: Quick ray emission demo

Page 32: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Intersection Processor Arrayor

“Who needs Fixed Function?”

Page 33: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Ray:AABB Test

• “Fast Ray-Axis Aligned Bounding Box Overlap Tests with Plucker Coordinates.” – Jeffrey Mahovsky and Brian Wyvill

• 6 lines form the silhouette of the AABB

• 6 planes from the ray origin and each edge vector

• Dot product of plane normal and ray direction vector

• 6 signs must match and be negative.

Page 34: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

3 MUL 6 SUB 6 ADD 12 MUL 6 bitwise OR

!voxel_center_x

origin_xcov_convertAX0min

BX0max

shift_center_x7+/!

ri(31)

XB

XA

ri(31)

XA

XB

>70

invert

ri(31)

>7

invert

ri(31)

ri

!voxel_center_y

origin_ycov_convertAY0min

BY0max

shift_center_y7+/!

rj(31)

YB

YA

rj(31)

YA

YB

>70

invert

rj(31)

>7

invert

rj(31)

rj

!voxel_center_z

origin_zcov_convertAZ0min

BZ0max

shift_center_z7+/!

rk(31)

ZB

ZA

rk(31)

ZA

ZB

>70

invert

rk(31)

>7

invert

rk(31)

rk

MISS

origin_x_miss

endpt_x_miss

origin_y_miss

endpt_y_miss

origin_z_miss

endpt_z_miss

Octant(0)

Octant(1)

Octant(2)

!

rj

rk

rixdirx

ray_distance

xdiry

ray_distance

xdirz

ray_distance

!

Ray distance setup Translation Box Plane Setup Plane tests Test Combine

Hitor

Miss

Page 35: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

USC instruction group packingMULMUL

MULSUB

LDLD

LDLD

LDLD

SUBSUB

SUB–

SUBSUB

ADDADD

ADDADD

ADDADD

MULMUL

MULMUL

MULMUL

MULMUL

MULMUL

MULMUL

OR–

OR–

OR–

26 Instructions

(32 if using compressed data formats)OR

OR–

OR–

Page 36: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

44x less area for this function

Tester and Rays

ChildAddress

Tester and Rays

Triangle vs. Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Tester and Rays Tester and Rays

Triangle vs. Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Triangle vs. Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Triangle vs. Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Boxvs.

Ray

Coherence Queue RAM

Old Count Existing Queue

Update Existing Queue

Update New

Queue

Coherence Cache

2 7 2 4 2 2

2 2 1 5

5 2 3 2 1 2

5 3 1 2

5 2 7 7 2 5

Queue Packer

Inst Data Inst Data

BoxHits

FreeQueue

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

USC

Page 37: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

The Ray Tracing Unitand

Coherence Engine

Page 38: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

System Memory Bus

System Memory Interface

Course Grain

Scheduler

RayData Master

USC0 USC1Texture Unit

USCn-1 USCnTexture Unit

Unified Shading Cluster Array

Intersection Processor Array Coherency Engine

Ray Tracing Unit

Rays in arbitrary orderRays grouped by PrimitiveObject

(Shader + Resources)

Page 39: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

( around 100Mbytes for 1M triangles including varyings )

AABB Blocks Vertex Blocks Varyings

Ascending Virtual Memory Addresses

Page 40: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

PrimitiveObject 0

PrimitiveObject 1

PrimitiveObject 2

Page 41: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

node_22359node_22355

node_16991

node_6445

node_6450

node_16994

node_6452

node_6456

node_16999 node_6436

node_6438 node_6440

node_11744node_6442

node_22372node_22369

node_6428

node_11722node_6430 node_6432

node_11726node_6434

node_17005node_11732 node_6424

node_11730node_6426

node_22409node_22404

node_11714

node_16978node_11716 node_16981node_11718

node_22418

node_16984

node_16986

node_17121

node_22423

node_14127

node_13585

node_13629 node_17200node_13503

node_22510

node_22517 node_22528

node_22533

node_22543

node_22550

node_22558

node_13707

node_19064

node_7590

node_13788node_7608

node_7638

node_13868node_7676 node_7696

node_13977node_7714

node_22561

node_19307 node_8136

node_14009node_8290 node_8362

node_8419

node_19446 node_8774

node_9065 node_9116

node_9228

node_22490node_22487

node_13993

node_19252

node_13994

node_8261

node_8356

node_8412

node_19374 node_8725

node_9046 node_9110

node_9221

node_22497

node_22502

node_21187

node_18926 node_7854

node_8219 node_8344

node_8394

node_19067 node_8549

node_8998 node_9102

node_14080node_9213

node_21192

node_7502

node_13643node_7544 node_7586

node_13722node_7606

node_19311node_13887 node_7622

node_13802node_7662

node_21196

node_21202

node_21326

node_21331 node_21338

node_21343

node_21663node_21658

node_13393

node_16862node_13455 node_17007node_13523

node_21672

node_17123

node_17203

node_19703node_17454

node_21677

node_17503node_11809

node_11822

node_17555node_11835

node_21822

node_21828

node_11843

node_17816

node_6602

node_11863node_6604

node_6594

node_11883node_6596 node_6598

node_11903node_6600

node_21834

node_21839

node_17935 node_6620

node_11926node_6624 node_6630

node_6634

node_17998 node_6608

node_6610 node_6612

node_6616

node_21237

node_21242

node_21249

node_18928

node_18933

node_7722

node_7738

node_8174

node_8182

node_18937

node_8332

node_8336

node_8374

node_8378

node_18944

node_7766

node_8194

node_18948

node_8340

node_8390

node_19069

node_19071

node_19077

node_8439

node_8808

node_8447

node_8451

node_8895

node_8899

node_19082

node_9078

node_9082

node_9086

node_9126

node_14074

node_9159

node_9163

node_19091

node_8520

node_8919

node_19094

node_9098

node_14083node_9184

node_21253

node_19203

node_19206

node_7478

node_13633node_7520

node_13637 node_7482

node_7486 node_7524

node_7528

node_19210

node_7562

node_13712node_7594

node_13716node_7598 node_7566

node_7570

node_19217 node_13645

node_7498

node_13649node_7540

node_13724

node_7582

node_13728node_7602

node_19313node_13885

node_19317

node_7610

node_13792node_7650 node_7614

node_13796node_7654

node_19321

node_13879 node_13870

node_13874

node_19329

node_13804

node_7618

node_13808node_7658

node_19331

node_13890

node_13892

node_21260

node_21270 node_21278

node_21284

node_21685

node_21692

node_13389

node_16866node_13451

node_16869

node_13379

node_13383

node_16873

node_13441

node_13445

node_16879node_13395

node_16882node_13457

node_17010

node_13515

node_13517

node_17013node_13591

node_17017

node_13505

node_13509

node_17021node_13587

node_17026node_13525

node_17029node_13595

node_21702

node_11756

node_17262node_11760

node_17265

node_12258

node_12260

node_17269

node_12347

node_12349

node_17274node_12254

node_17277node_12345

node_11764

node_17308node_11768

node_17311node_12270

node_17315node_12266

node_12354

node_17324node_12358

node_21713

node_17255

node_11749

node_17244node_11752

node_17246node_12246

node_17249node_12242

node_19706

node_11654

node_17127node_11688

node_17136

node_17129node_11650

node_17132node_11684

node_17206node_11747

node_17210node_11745

node_17222node_17218node_12236

node_17229

node_17231

node_17234node_12336

node_12421

node_17458node_12500 node_17459

node_17463node_12417

node_17466node_12492

node_21730

node_21736

node_17507node_12594

node_17509node_12622

node_17514node_11803

node_17516node_12590

node_17521

node_11811

node_11814

node_17525

node_12616 node_12608

node_12611

node_17558node_12646

node_17561node_12676

node_17566node_11824

node_17569node_11837

node_17574

node_12636

node_12640

node_17579

node_12666

node_12670

node_21746

node_21751

node_21766node_21760

node_17388

node_17390

node_17396

node_6655

node_6657

node_6985

node_6987 node_6993

node_6995

node_17401

node_6650

node_6964

node_6966

node_6652

node_12320

node_6980 node_6972

node_6974

node_17411

node_7109

node_7111

node_17414

node_7105

node_12395node_7107

node_17422

node_17427

node_6673

node_6678

node_7009

node_7012

node_17434

node_17437

node_6661

node_6671

node_7003

node_7005

node_17445

node_7125

node_7131

node_17448

node_7115

node_7117

node_21782node_21777

node_17328

node_17331node_11772

node_17336

node_6638

node_12288node_6934 node_6640

node_12292node_6936

node_17340

node_12369

node_7093

node_12373node_7095

node_17345

node_12282 node_12274

node_12277

node_17352

node_12362

node_12365

node_17361

node_17366

node_6646

node_12308

node_6948

node_6950

node_6648

node_12312

node_6956

node_6958

node_17371

node_6642

node_12298

node_6938

node_6940

node_6644

node_12302node_6946

node_17377

node_12385

node_7101

node_12389node_7103

node_17380

node_12377

node_7097

node_12381node_7099

node_21890

node_21897

node_17818

node_11846

node_11848

node_17821

node_11869 node_6491

node_11867node_6493

node_17828node_17825node_12706

node_17833

node_12692

node_12696

node_12699

node_17840

node_7151

node_12724node_7153 node_6729

node_12728node_6731

node_17877

node_17883

node_11889 node_6495

node_11887node_6499

node_11909 node_6503

node_11907node_6507

node_17886

node_7155

node_12748node_7159 node_6733

node_12752

node_6743

node_6747

node_17890

node_7163

node_12771node_7167

node_12775 node_6759

node_6763 node_6775

node_6779

node_21904

node_21910

node_17939 node_6511

node_11929node_6517 node_6521

node_6525

node_17941

node_17944

node_17950

node_12795node_7171 node_6785

node_6789

node_7178 node_6804

node_6808

node_17957

node_7182

node_7186

node_6820

node_6824 node_6836

node_6840

node_18000

node_18003

node_18009

node_6530

node_6538

node_18012

node_6546

node_6558

node_18017

node_6871

node_7191

node_7202

node_6857

node_6861

node_18023

node_6877

node_6883

node_7220

node_7226

node_21917

node_21924 node_21933

node_21938

node_21354

node_21360 node_21369

node_21374

node_21489

node_21496

node_19161 node_19017

node_19020

node_21503

node_21509

node_19283

node_19413node_13947

node_21969

node_21977

node_18068

node_5839

node_5842

node_18071

node_5844

node_5846

node_18146 node_5814

node_5820 node_5824

node_11979node_5832

node_21983

node_21989

node_5852

node_11988node_5854 node_5856

node_12011node_5858

node_18275node_12049 node_5848

node_12031node_5850

node_22102

node_22108

node_12072

node_18344node_12080 node_18591node_12093

node_22115

node_13427

node_16946node_13489 node_13565

node_17092node_13617

node_22120

node_18640

node_19784node_18688

node_11674

node_11705

node_22041

node_22047

node_18074

node_18077

node_18082

node_5630

node_5644

node_18087

node_5650

node_5656

node_18092

node_5889

node_5904

node_6246

node_6259

node_18099

node_5910

node_5916

node_6265

node_6272

node_18149

node_5669

node_5674

node_18151

node_5678

node_11982node_5682

node_18155

node_18157

node_18164

node_6290

node_6295

node_5931

node_5935 node_5952

node_5956

node_18170

node_6299 node_5968

node_5972

node_12869node_6303 node_5984

node_5988

node_22053

node_18215

node_18221

node_11994 node_5687

node_11992node_5691

node_12017 node_5695

node_12015node_5699

node_18224

node_6307

node_12883node_6311

node_12887 node_6000

node_6004 node_6010

node_6014

node_18229

node_6315

node_12908node_6319

node_12912node_6042 node_6026

node_6030

node_18276

node_18279node_12964

node_18284

node_12037 node_5703

node_12035node_5705

node_18286

node_12060 node_12053

node_12056

node_18292

node_6323

node_12932node_6325 node_6052

node_12936node_6054

node_18297

node_12960 node_12953

node_12955

node_22073 node_22057

node_22066

node_21516

node_21521

node_21530

node_19024

node_19028

node_19034

node_8784

node_9071

node_19039

node_9122

node_9234

node_19044

node_8759

node_9059

node_9067

node_8776

node_8780

node_19052

node_9114

node_9118

node_9226

node_9230

node_19166

node_8150

node_8307

node_19168

node_8372

node_8434

node_14121node_8436

node_19172

node_19175

node_19180

node_8061

node_8288

node_8145node_8140

node_8294

node_8298

node_19186

node_8360

node_8364

node_8368

node_8416

node_14115

node_8429 node_8421

node_8425

node_21533

node_19285

node_19289

node_13703 node_7648

node_13701node_7684

node_13782 node_7702

node_13780node_7720

node_19292

node_13691 node_7640

node_7644 node_7683node_7678

node_7636

node_13695node_7674

node_19296

node_7698

node_13770node_7716 node_7694

node_13774node_7712

node_19416node_13949

node_19420

node_13862 node_7518

node_13860node_7560

node_19422

node_13971 node_13964

node_13967

node_19427

node_7514

node_13850node_7556 node_7512

node_13854node_7554

node_19431

node_13960

node_13955node_13953node_7592

node_21540

node_21548 node_21557

node_21562

node_22191

node_12982

node_18376node_13000

node_18379node_12074

node_18382node_12082

node_18387node_12978

node_18391node_12996

node_18595node_13024

node_18598node_13040

node_18602

node_12095

node_12097

node_18606

node_13017

node_13019

node_18612node_12106

node_18614node_13036

node_22194

node_22201

node_13241

node_18745node_13245

node_18748node_12177

node_18751

node_13282

node_13286

node_13289

node_18758node_12175

node_18760node_13278

node_13249

node_18778node_13253

node_18781node_12185

node_18784node_12181

node_18789

node_13305

node_13309

node_18794

node_13295

node_13299

node_22210

node_13437

node_16950node_13499

node_16953node_13429

node_16956node_13491

node_16961node_13433

node_16966node_13495

node_17094

node_13578

node_13580

node_17097node_13627

node_17101node_13567

node_17104node_13619

node_17108

node_13571

node_13574

node_17113node_13623

node_22218

node_13227

node_18727node_13233 node_18728

node_18732node_13270

node_18735node_13262

node_13064

node_18644node_13182 node_18645

node_18649node_13056

node_18652node_13175

node_17181

node_11676

node_17186node_11707

node_17188node_11680

node_17191node_11710

node_19786

node_13225

node_18690

node_18693node_13223

node_18699

node_18706node_13260

node_18720

node_18714node_12173

node_22236

node_22242

node_22251

node_18863

node_18868

node_5790

node_5792

node_18870

node_5788

node_5785

node_12226node_5883

node_18875

node_6403

node_6405

node_6208

node_6212 node_6221node_6216

node_18881

node_6401 node_6200

node_6204

node_13363node_6399 node_6192

node_6196

node_18894

node_18899

node_5801

node_5805

node_18902

node_5794

node_5799

node_18906

node_6232

node_6419

node_6421

node_6238

node_6242

node_18914

node_6222

node_6228

node_6407

node_6417

node_22256

node_18804

node_18807node_13257

node_18812

node_12204 node_5773

node_12202node_5775

node_18815

node_6166

node_13331node_6389 node_6162

node_13335node_6387

node_18821

node_12196 node_12189

node_12192

node_18825

node_13323 node_13315

node_13319node_6160

node_18836

node_18841

node_12220 node_5781

node_12218node_5783

node_18844

node_12212 node_5777

node_12210node_5779

node_18849

node_13351node_6397 node_6184

node_6188

node_13355node_6395node_6183node_6178

node_18854

node_6174

node_13341node_6393 node_6170

node_13345node_6391

node_22290

node_22294

node_22300

node_16820

node_16823node_10766

node_22307

node_22311 node_22317

node_22323

node_22431

node_22438

node_11100

node_16838node_11102 node_11104

node_16841node_11106

node_22446

node_10782

node_16827node_10804 node_10833

node_16831node_10853

node_22451

node_11094

node_19643

node_10965

node_11031

node_11096

node_11098

node_22459

node_22467

node_16854

node_16857

node_22474

node_22480

node_16850 node_16843

node_16846node_11108

node_21211

node_21216

node_21225

node_19254

node_19259

node_7692

node_13760node_7710 node_7688

node_13764node_7706

node_19264

node_7634

node_13681node_7672

node_13685node_7670 node_7626

node_7628

node_7671

node_7666

node_19272

node_13750

node_7686

node_13754node_7704

node_19275

node_13671

node_7624

node_13675node_7664

node_19380node_19377node_13945

node_19385

node_13933

node_13935

node_13939node_7588

node_19394

node_7510

node_13840node_7552 node_7506

node_13844node_7548

node_19403

node_13922

node_13924

node_13926

node_19406

node_13830

node_7504

node_13834node_7546

node_21229

node_18988

node_18993

node_9106

node_9112

node_9217

node_9224

node_18999

node_8744

node_9042

node_9053

node_8717

node_8719

node_19006

node_9104

node_9215

node_19009

node_8715

node_9040

node_19131

node_19136

node_14105node_8414 node_8399

node_8402

node_8404

node_8358 node_8348

node_8350

node_19145

node_7960

node_7995

node_8286

node_7961

node_7956

node_8251

node_8253

node_19157 node_7950

node_8245 node_8346

node_14101node_8396

node_21293

node_21300 node_21311

node_21316

node_21598

node_14145

node_14139 node_14131

node_14134

node_18661

node_13008

node_13011

node_18665

node_12086

node_12088

node_18671node_12990

node_18674node_12078

node_18680node_13004

node_18683node_12986

node_12102

node_12108

node_13032

node_13048

node_13028

node_13044

node_21608

node_18630

node_18623node_13274

node_18626node_13237

node_13230

node_18637node_13266

node_19762

node_17174

node_11670

node_17163node_11702

node_17165node_11698

node_17168node_11666

node_13185

node_18446node_18444node_13187

node_18457

node_18458

node_18453node_12112

node_18465node_18460node_13068

node_13171

node_18427node_13179

node_18436

node_18429node_13060

node_18432node_13052

node_21623

node_21630

node_13189

node_18348node_13193

node_18351

node_13080

node_13084

node_18356

node_13070

node_13074

node_18363node_12118

node_18366node_12114

node_13197

node_18402node_13201

node_18405node_13103

node_18408node_12126

node_18412

node_13097 node_13090

node_13093

node_18417node_12122

node_21641

node_21647

node_13423

node_16922node_13485

node_16925node_13481

node_16930node_13419

node_13415

node_16940node_13477

node_17066node_13613

node_17068

node_13557

node_13561

node_17073node_13609

node_17076

node_13550

node_13553

node_13546

node_17086node_13607

node_21846

node_18240

node_6301

node_6305

node_5976

node_5978 node_5992

node_5994

node_18244

node_6297 node_5960

node_5962

node_12843node_6292 node_5939

node_5942

node_5946

node_18254

node_18256

node_18265

node_5680

node_5685

node_18267

node_5676

node_5671

node_11961node_5837

node_18311

node_5912

node_5920

node_6267

node_6274

node_18318

node_5892

node_5906

node_6249

node_6261

node_18323

node_18327

node_18334

node_5652

node_5658

node_18337

node_5633

node_5646

node_21854

node_18111

node_6056

node_12918node_6058 node_6327

node_12922node_6329

node_18116

node_12894

node_12898

node_18125

node_12021

node_5707

node_12025node_5709

node_18128

node_12005 node_11998

node_12000

node_18135

node_18138node_12891

node_18207

node_18185

node_12968 node_6034

node_6036 node_6044

node_6046

node_6317

node_12972node_6321

node_18191

node_6008

node_12942

node_6018

node_6020

node_6309

node_12946node_6313

node_18198

node_12064

node_5697

node_12068node_5701

node_18201

node_12041

node_5689

node_12045node_5693

node_21862

node_21867

node_21876

node_21881

node_21571

node_21577

node_18531

node_18536

node_6090

node_13119node_6361 node_6086

node_13123node_6359

node_18540

node_6084

node_13109node_6357

node_6085

node_6080

node_13113node_6355 node_6072

node_6076

node_18547

node_12136

node_5721

node_12140node_5723

node_18550

node_12128

node_5717

node_12132node_5719

node_18561node_18559node_13205

node_18566

node_13137

node_13141

node_13145node_6070

node_18572

node_12152

node_12154

node_12156

node_18578

node_6066

node_13129node_6353 node_6062

node_13133node_6351

node_18582

node_12144

node_5713

node_12148node_5715

node_21581

node_21589

node_18476

node_18481

node_6134

node_6138

node_6383

node_6385

node_18486

node_6124

node_6130

node_6371

node_6373

node_18494 node_5734

node_5736 node_5741

node_5743

node_18500

node_18505

node_13165node_6369 node_6116

node_6120

node_6367 node_6108

node_6112

node_18513

node_6098

node_6363

node_6365

node_6099

node_6094

node_6100

node_6104

node_18521

node_5729

node_12171node_5731

node_18524

node_5725

node_5727

node_21793

node_21799 node_21808

node_21813

node_21383

node_21388 node_21396

node_21402

node_21447node_21442

node_19244

node_19224

node_13742node_7600 node_7574

node_7576

node_7584

node_13746node_7604

node_19228

node_13663 node_7490

node_7492 node_7532

node_7534

node_7500

node_13667node_7542

node_19235

node_13732

node_7564

node_13736node_7596

node_19238

node_13653

node_7480

node_13657node_7522

node_19340

node_13908

node_13911

node_13915

node_19346

node_7616

node_13822node_7656 node_7620

node_13826node_7660

node_19352

node_13897

node_13899

node_19356

node_13812

node_7612

node_13816node_7652

node_19366node_19363node_13904

node_21463node_21455

node_18982

node_18958

node_8338

node_8342

node_8392 node_8382

node_8384

node_18963

node_7752

node_7768

node_8188

node_8196

node_18972

node_8334

node_8376

node_18975

node_7724

node_8176

node_19103

node_14091node_9190 node_9167

node_9170

node_9172

node_9100 node_9090

node_9092

node_19110

node_8526

node_8929

node_8486

node_8488

node_8903

node_8905

node_19116

node_19119

node_19127 node_8441

node_8818 node_9080

node_14087node_9133

node_21475

node_21480

node_22002node_21997

node_17990

node_17970

node_12734node_6783 node_6767

node_6769

node_7165

node_12738node_7169

node_17974

node_12710 node_6735

node_6737 node_6751

node_6753

node_7157

node_12714node_7161

node_17981

node_11873

node_6505

node_11877node_6509

node_17984

node_11853

node_6497

node_11857node_6501

node_18034

node_12782

node_12784

node_12787

node_18041

node_11913

node_11915

node_11917

node_18047

node_6725

node_12758node_6727 node_7147

node_12762node_7149

node_18051

node_11893

node_6487

node_11897node_6489

node_18061node_18058node_12779

node_22018node_22011

node_17850

node_6879

node_7222

node_7228

node_6885

node_6887

node_17854

node_6865

node_6873

node_7193

node_7206

node_17859

node_17862

node_17871 node_6532

node_6540 node_6548

node_6560

node_17900

node_12829node_7188 node_6844

node_6847

node_6849

node_7184 node_6828

node_6830

node_17908

node_7176

node_7180

node_6796

node_6798 node_6812

node_6814

node_17914

node_17917

node_17925

node_6523 node_6527

node_11951node_6614

node_17928

node_6515

node_6519

node_22028

node_22032

node_22129

node_19744

node_17151

node_17144node_11694

node_17147node_11662

node_11658

node_17158node_11691

node_17653

node_17657node_12504

node_17664node_11783

node_17670node_11781

node_17684

node_17680node_17678node_12424

node_17543

node_17536node_12341

node_17539node_12250

node_12238

node_17550node_12338

node_17648

node_17636node_12496

node_17639node_12488

node_11774

node_17645node_11778

node_22143

node_16891

node_13469

node_13473

node_16895

node_13407

node_13411

node_16901node_13461

node_16904node_13399

node_16911node_13465

node_16914node_13403

node_17037node_13603

node_17040node_13599

node_17044

node_13537

node_13541

node_17049node_13533

node_17057node_13597

node_17059node_13529

node_22163node_22156

node_17590node_12510

node_17594node_12506

node_11785

node_17601node_11789

node_12426

node_17607node_12430

node_17612node_12522

node_17615node_12441

node_17619

node_12514

node_12518

node_17623

node_12434

node_12437

node_11793

node_17631node_11797

node_22180node_22174

node_17285node_12630

node_17289node_12602

node_11805

node_17296node_11818

node_12598

node_17302node_12626

node_17475node_12684

node_17478node_11841

node_17482

node_12658

node_12662

node_17487

node_11828

node_11831

node_17494node_12680

node_17496

node_12650

node_12654

node_21413

node_21419 node_21426

node_21431

node_21947

node_21950 node_21955

node_21960

node_22082node_22079

node_17774

node_17754

node_6692

node_12538node_7034 node_6694

node_12542

node_7036

node_7038

node_17758

node_6688

node_12528

node_7018

node_7020

node_6690

node_12532

node_7026

node_7028

node_17765

node_12451

node_6905

node_12455node_6907

node_17768

node_12443

node_6901

node_12447node_6903

node_17781

node_12556

node_12558

node_12561

node_17789

node_12467

node_12469

node_12471

node_17795

node_6684

node_12548node_7014 node_6686

node_12552node_7016

node_17799

node_12459

node_6897

node_12463node_6899

node_17808node_17806node_11801

node_22094node_22092

node_17711

node_17691

node_6719

node_6723

node_7085

node_7091

node_17696

node_6707

node_6709

node_7081

node_7083

node_17702

node_6928

node_6930

node_17705

node_6918

node_6920

node_17718

node_6700

node_12582

node_7060

node_7063

node_7065

node_6703

node_7073

node_7075

node_17728

node_6696

node_6698

node_7044

node_7046 node_7052

node_7054

node_17735

node_17738

node_17745

node_6913

node_12486node_6916

node_17748

node_6909

node_6911

node_22274 node_22263

node_22268

node_22279

node_22282

node_22332

node_22335 node_22340

node_22345

node_22381

node_22384 node_22389

node_22393

node_19955

node_19959

node_19964

node_19975node_19974

node_12822node_6345

node_19984node_19981

node_6335

node_12690node_6337 node_6339

node_12720node_6341

node_16792node_12766 node_6331

node_12744node_6333

node_20031

node_20037

node_16766

node_16773node_16771node_12403

node_20042

node_20047

node_16755

node_16759

node_16762

node_20056

node_12588

node_16748node_12606 node_16751node_12634

node_20061

node_16753

node_19624node_16786 node_12413

node_12415

node_20067

node_20073

node_12405

node_16779node_12407 node_12409

node_16783node_12411

node_19814node_19804

node_16554

node_9822

node_9826

node_9828

node_16559node_9756

node_16564node_9835

node_16567node_9760

node_9752

node_16574node_9818

node_16693

node_9962

node_9966

node_16698

node_9898

node_9902

node_16705node_9972

node_16708node_9908

node_9894

node_16716node_9958

node_19831

node_19838

node_19853

node_19858

node_19867

node_11473

node_15940node_11477

node_15943

node_10927

node_10931

node_15947

node_10919

node_10923

node_15953node_11012

node_15956node_11008

node_15987node_11489

node_15989node_10947

node_15994

node_11481

node_11485

node_15997

node_10942 node_10935

node_10938

node_16006node_11023

node_16008

node_11016

node_11019

node_16465

node_16458node_11638

node_16461node_11605

node_11602

node_16472node_11634

node_19518

node_16037node_16033node_11506

node_16042node_11504

node_16045node_10963

node_16060 node_16053

node_16056node_11029

node_16107

node_11515

node_16096node_11551

node_16098node_11090

node_16101node_11215

node_16027

node_11493

node_16016node_11501

node_16018node_10959

node_16021node_10955

node_16134node_11428

node_16136node_10778

node_16141node_11438

node_16144

node_10796

node_10800

node_16151node_10776

node_16153node_10792

node_16385node_11452

node_16388node_11463

node_16393

node_10825

node_10829

node_16397

node_10845

node_10849

node_16405node_10821

node_16408node_10841

node_20843

node_14528

node_14503

node_6142

node_6349

node_6144

node_6146

node_6375

node_6377

node_14508

node_5865

node_5867

node_5869

node_5745

node_9270

node_5747

node_5750

node_5754

node_14518

node_6236

node_6454

node_14521

node_5803

node_9276node_5887

node_14665

node_14646

node_6869

node_6875

node_7198

node_7218

node_14650

node_6536

node_6542

node_6622

node_6626

node_14656

node_6895

node_7232

node_14659

node_6564

node_6636

node_20850

node_14252

node_7298

node_9530node_7338 node_7296

node_9534node_7336

node_14257

node_9442 node_9433

node_9437

node_14267

node_9538

node_7304

node_9542node_7344

node_14269

node_9455 node_9448

node_9450

node_14276

node_14279node_9430

node_14398

node_14381

node_9690 node_7416

node_7418 node_7458

node_7460

node_7414

node_9694node_7456

node_14386

node_7354

node_9610

node_7374

node_7376

node_7352

node_9614node_7372

node_14394 node_9618

node_7360

node_9622node_7392

node_9698

node_7434

node_9702node_7476

node_20858

node_20864

node_20873

node_20877

node_20363

node_20368

node_20376

node_15593node_11417

node_15598

node_10644

node_10646 node_10653node_7934

node_15605

node_15608

node_8241

node_10685node_8243 node_7940

node_10689node_7946

node_15617

node_10637

node_10639

node_15621

node_10675

node_7938

node_10679node_7944

node_15666

node_15671

node_8223

node_10721node_8227 node_7874

node_10725node_7888

node_15676

node_8231

node_10758node_8235 node_7904

node_10762

node_7905

node_7900

node_7914

node_7916

node_15684

node_10711

node_7870

node_10715node_7884

node_15687

node_10748

node_7896

node_10752node_7910

node_20384

node_15729

node_15734

node_10569node_8204 node_7794

node_7796

node_8209 node_7814

node_7816

node_15743

node_7860

node_8213

node_8217

node_7834

node_7836

node_7861

node_7856

node_15751

node_7789

node_10565node_7810

node_15754

node_7830

node_7850

node_15793

node_15796

node_7734

node_7748

node_8180

node_8186

node_15802

node_7762

node_7779

node_8192

node_8200

node_15810

node_7730

node_7744

node_15813

node_7758

node_7775

node_20394

node_20399

node_20947

node_20953 node_20960

node_20965

node_20188

node_20193

node_20200

node_15882

node_15885

node_8047

node_10887node_8278

node_10891node_8280 node_8049

node_8051

node_15889

node_10877node_8274 node_8031

node_8033

node_10881node_8276 node_8039

node_8041

node_15896

node_10985

node_8166

node_10989node_8168

node_15899

node_10977

node_8162

node_10981node_8164

node_15908node_11469

node_15912

node_10905

node_10908

node_10912

node_15919

node_11001

node_11003

node_15926

node_8057

node_10897node_8282 node_8059

node_10901node_8284

node_15930

node_10993

node_8170

node_10997node_8172

node_20208

node_15825

node_15830

node_7991

node_7997

node_8259

node_8263

node_15835

node_7987

node_7989

node_8247

node_8249

node_15841

node_8148

node_8152

node_15844

node_8138

node_8146

node_15852

node_15855

node_15860

node_10869node_8269 node_8015

node_8017

node_8272 node_8023

node_8025

node_15868

node_8265

node_8267

node_7999

node_8001 node_8007

node_8009

node_15878 node_8154

node_8156 node_8158

node_10975node_8160

node_20216

node_20222

node_20334

node_20340 node_20348

node_20353

node_20813

node_20819 node_20828

node_20833

node_20540

node_20546

node_15320

node_15323

node_15328

node_8753

node_9050

node_9057

node_8734

node_8738

node_15336

node_8768

node_8797

node_9063

node_9076

node_15344

node_8727

node_8747

node_15347

node_8762

node_8787

node_15391

node_15396

node_8653

node_9026

node_9030

node_8654

node_8649

node_8667

node_8669

node_15403

node_9034 node_8687

node_8689

node_10553node_9038 node_8705

node_8707

node_15417 node_8639

node_8657 node_8677

node_10547node_8697

node_20554

node_20560

node_15461

node_15466

node_9010

node_10382node_9014

node_10386node_8605 node_8585

node_8587

node_8606

node_8601

node_15471

node_9018

node_10418node_9022 node_8617

node_10422node_8633

node_15481 node_10372

node_8579

node_10376node_8595

node_10408

node_8609

node_10412node_8623

node_15526

node_15529node_11415

node_15534

node_9000

node_10454node_9002 node_8553

node_10458node_8561

node_15539

node_10491node_10488node_8569

node_15549

node_10444

node_8551

node_10448node_8557

node_15551

node_10480

node_10482

node_20567

node_20573

node_20681

node_20688

node_11374

node_14939node_11384

node_14942node_10278

node_14946node_10304

node_14952node_10274

node_14955node_10300

node_14982node_11394

node_14985

node_10324

node_10327

node_14992node_11404

node_14994node_10345

node_15001node_10320

node_15004node_10343

node_20697

node_9740

node_16533node_9804

node_16536node_9814

node_16541node_9748

node_9746

node_16549node_9810

node_9880

node_16670node_9946

node_16673node_9954

node_16678node_9890

node_9886

node_16688node_9950

node_20709

node_15052

node_11287

node_15041node_11293

node_15043node_10092

node_15046node_10127

node_16452

node_11564

node_16441node_11626

node_16443node_11630

node_16446node_11598

node_19497

node_11301

node_15248node_15246node_10161

node_15260node_15255node_11366 node_15264

node_15270node_10266

node_10150

node_15229node_10158 node_15230

node_15234node_10262

node_15237node_10254

node_20726

node_20732

node_11348

node_15175node_11352

node_15178node_10138

node_15181node_10134

node_15186

node_10234

node_10238

node_15191

node_10226

node_10230

node_11356

node_15205node_11360

node_15208node_10146

node_15210node_10250

node_15215node_10142

node_15218

node_10242

node_10246

node_21003

node_21010 node_21018

node_21023

node_21082

node_21087

node_21095

node_14465

node_14467

node_14474

node_6682

node_7089

node_7123

node_6711

node_6713

node_7124

node_7119

node_14480

node_6580

node_6584

node_6586

node_6472

node_9262

node_6477

node_6479

node_14493

node_6705

node_7113

node_14496

node_6475

node_9256node_6582

node_14615

node_14620

node_5914

node_5925

node_6269

node_6280

node_6284

node_14626

node_5654

node_5663

node_5822

node_5828

node_14634

node_5923

node_6277

node_14637

node_5661

node_5826

node_21104

node_14218

node_14221node_9409

node_14226

node_7402

node_9520node_7444 node_7406

node_9524node_7448

node_14231

node_9422

node_9426

node_9423

node_7364

node_14240

node_9510

node_7404

node_9514node_7446

node_14243

node_9415

node_9417

node_14351

node_14356

node_7280

node_9680node_7318 node_7288

node_9684

node_7289

node_7284

node_7322

node_7324

node_14362

node_7242

node_9600node_7260 node_7246

node_9604node_7264

node_14369

node_9670

node_7282

node_9674node_7320

node_14372

node_9590

node_7244

node_9594node_7262

node_21114

node_21117

node_20507 node_20493

node_20499

node_20628

node_20636

node_15119

node_15125

node_11321 node_9201

node_11319node_9203

node_11329 node_9205

node_11327node_9207

node_15128

node_8510

node_10195node_8984 node_8508

node_10199node_8980

node_15132

node_8506

node_10185

node_8979node_8974

node_8504

node_10189

node_8966

node_8970

node_15142

node_11341

node_11343

node_15145

node_11337 node_9209

node_11335node_9211

node_15149node_10131

node_15153

node_10213

node_10216

node_10220node_8996

node_15161

node_15164

node_8514

node_10205node_8992 node_8512

node_10209node_8988

node_20642

node_20647

node_15060

node_15066 node_9180

node_9182 node_9186

node_9188

node_15069

node_8484

node_8494

node_8921

node_8925

node_15075

node_8480

node_8482

node_8911

node_8915

node_15086

node_15089

node_15094

node_11311

node_9197

node_11312

node_9199

node_15097

node_9193

node_9195

node_15101

node_8502

node_10177

node_8953

node_8956

node_8960

node_8500

node_8945

node_8949

node_15108

node_8496

node_8498

node_8936node_8931

node_8937

node_8941

node_21032

node_21037 node_21045

node_21050

node_19995

node_20002

node_7135

node_12879node_7137 node_7139

node_12904node_7141

node_16797node_12950 node_7143

node_12928node_7145

node_20008

node_12864node_7133

node_20026 node_20012

node_20019

node_20082

node_16816node_13217

node_13219

node_13221

node_20086

node_16805

node_16811

node_16813

node_20091

node_20099

node_12976

node_16800node_12994 node_16803node_13015

node_20141node_20137

node_13208node_7200

node_7204

node_7208

node_7210

node_7212 node_7214

node_7216

node_20154node_20152node_13215

node_20238

node_11378

node_15059node_11388 node_11398

node_15284node_11408

node_20243

node_11296

node_19475 node_11303

node_11368 node_11562

node_11624

node_20249

node_20256

node_9738

node_16528node_9801 node_9878

node_16640node_9944

node_20409

node_15485

node_15487 node_15557

node_15560

node_20413

node_15421 node_15355

node_15358node_11413

node_20418

node_20423

node_20886

node_20891

node_20901

node_14345node_14212node_9407

node_20904

node_14459

node_14608

node_20914

node_20920 node_20927

node_20936

node_20265

node_20271

node_14881

node_10102

node_10104

node_14884node_10098

node_14888

node_10057

node_10060

node_14893

node_11270

node_11272

node_14899node_10053

node_14902node_11268

node_10110

node_14911node_10114

node_14914

node_10074

node_10078

node_14919

node_10066

node_10070

node_14926node_11281

node_14929node_11277

node_20280

node_14152node_10339

node_14155node_10316

node_14160

node_10331

node_10335

node_14164

node_10308

node_10312

node_11390

node_14173node_11400

node_15012

node_10293

node_10295

node_15016

node_10289 node_10282

node_10284

node_15022node_10272

node_15024node_10268

node_15031node_11380

node_15034node_11370

node_20287

node_10120

node_14864node_10123

node_14873

node_14866node_10088

node_14869node_10084

node_14963

node_14967node_10258

node_14970node_10154

node_11298

node_14976node_11362

node_19458

node_11291 node_14829

node_14832node_10118

node_14843node_14839node_10082 node_14849

node_14852node_11285

node_16416

node_11559

node_16421node_11620

node_16423node_11555

node_16426node_11616

node_20303

node_16496node_9732

node_16498

node_9786

node_9788

node_16504node_9734

node_16506node_9728

node_16511

node_9793

node_9797

node_16515

node_9780 node_9772

node_9775

node_16607node_9870

node_16610node_9936

node_16615node_9874

node_16618node_9940

node_16623

node_9860

node_9864

node_16628

node_9926

node_9930

node_20308

node_20315

node_14765

node_14767

node_14773

node_8464

node_8466

node_8861 node_8849

node_8853

node_8862

node_8857

node_14779

node_8462

node_8841

node_8845

node_8460

node_9994

node_8836 node_8828

node_8832

node_14788

node_9143

node_9145

node_14790

node_9138

node_11225node_9141

node_14798

node_14803

node_8455

node_8457

node_8814

node_8820

node_8824

node_14811

node_8443

node_8445

node_8804

node_8810

node_14819

node_9131

node_9135

node_14822

node_9124

node_9129

node_20321

node_20327

node_14709

node_14712node_10096

node_14718

node_8478

node_10025node_8891 node_8476

node_10029node_8887

node_14722

node_11244

node_9155

node_11248node_9157

node_14726

node_10018 node_10010

node_10014node_8885

node_14734

node_11240 node_11233

node_11235

node_14743

node_14748

node_8474

node_10045

node_8877

node_8881

node_8472

node_10049node_8875

node_8876

node_8871

node_14752

node_8470

node_10035node_8867 node_8468

node_10039node_8863

node_14759 node_11252

node_9147

node_11256node_9149

node_11260

node_9151

node_11264node_9153

node_20432

node_20440

node_20448

node_15427 node_8675

node_10503

node_10504

node_8655

node_8695

node_8713

node_15430

node_9032

node_9036

node_8679

node_8683 node_8704node_8699

node_15437

node_9028 node_8659

node_8663

node_10499node_9024 node_8641

node_8645

node_15448

node_15451

node_15494

node_8772

node_8801

node_15497

node_8742

node_8757

node_15502

node_8764

node_9061

node_9073

node_8789

node_8793

node_15507

node_8730

node_8749

node_9048

node_9055

node_15514

node_15517

node_20453

node_15287

node_10404 node_8571

node_10402node_8577

node_15290

node_10368 node_10361

node_10364

node_15296

node_8565

node_10392node_8573 node_9004

node_10396node_9006

node_15301

node_10356

node_10352node_10350node_8559

node_15311

node_15314node_11410

node_15385

node_15363

node_10476 node_8621

node_10474node_8637

node_15366

node_10440 node_8593

node_10438node_8607

node_15371

node_10464 node_8616node_8611

node_8625

node_8629

node_9016

node_10468node_9020

node_15377

node_8581

node_10428node_8597 node_9008

node_10432node_9012

node_20460

node_20468 node_20478

node_20484

node_14178node_9405

node_14182

node_9398

node_9401

node_14185

node_9502 node_7398

node_9500node_7440

node_14189

node_9384

node_9391node_9389node_7362

node_14195

node_7394

node_9490node_7436 node_7400

node_9494node_7442

node_14321

node_14325

node_9582 node_7238

node_9580node_7256

node_9662 node_7276

node_9660node_7314

node_14328

node_7234

node_9570node_7252 node_7240

node_9574node_7258

node_14332

node_9650 node_7275node_7270

node_7306

node_7310

node_7278

node_9654node_7316

node_14435 node_9244

node_6576

node_9245

node_6468

node_6676

node_6932

node_14437

node_14442

node_9238node_6470 node_6460

node_6464

node_6578 node_6568

node_6572

node_14448

node_6680

node_7001

node_6663

node_6667

node_6927node_6922

node_14574

node_14576

node_14582

node_5642

node_5816

node_14585

node_5902

node_6257

node_14590

node_5638

node_5648

node_5810

node_5818

node_14596

node_5898

node_5908

node_6253

node_6263

node_20514

node_15762

node_15822node_11422

node_20519

node_15627

node_15695

node_15698

node_20524

node_20530

node_20655

node_16414

node_19560node_16493

node_11508

node_11513

node_20660

node_16603node_9846

node_9916

node_16744node_9980

node_20672node_20668

node_11432

node_16068node_11442 node_11456

node_16114node_11467

node_20974

node_20979 node_20987

node_20993

node_21059

node_9470

node_14320

node_7292

node_9556node_7332

node_7348

node_9636node_7368 node_7410

node_9716node_7452

node_21062

node_14572 node_5711

node_9284node_5860 node_6060

node_6343

node_14707 node_6513

node_6606 node_6793

node_7174

node_21066

node_21071

node_20750node_20744

node_16332

node_11179

node_11183

node_16336

node_11524

node_11526

node_16341node_11175

node_16344node_11520

node_16350node_11076

node_16353node_11074

node_16359

node_11199

node_11203

node_16363

node_11189

node_11193

node_16369node_11084

node_16372node_11080

node_11532

node_16380node_11536

node_20758

node_16326

node_16314node_11219

node_16317node_11211

node_11544

node_16323node_11547

node_19543

node_16277node_11209

node_16283node_11542

node_16288

node_16292node_11540

node_16300

node_16305node_16303node_11088

node_11613

node_16477node_11646

node_16486

node_16479node_11609

node_16482node_11642

node_16115

node_16119node_11025

node_16122node_10951

node_11497

node_16128node_11510

node_20770

node_9768

node_16580node_9856

node_16583node_9764

node_16586

node_9848

node_9852

node_16592node_9762

node_16594

node_9839

node_9842

node_9922

node_16722node_9986

node_16725node_9918

node_16729node_9982

node_9912

node_16736node_9976

node_20781

node_10772

node_15966node_10788

node_15969node_10784

node_15973node_10768

node_11424

node_15980node_11434

node_16072node_10839 node_10814

node_10816

node_16075node_10835

node_16078node_11459

node_16082

node_10806

node_10810

node_16086

node_11444

node_11448

node_20791node_20787

node_16209

node_8316

node_8318

node_8090

node_8092 node_8098

node_8100

node_16214

node_8311 node_8074

node_8076

node_11114node_8313 node_8082

node_8084

node_16223

node_7971

node_7973

node_16226

node_7967

node_11035node_7969

node_16232

node_16234

node_16242

node_8069

node_8072

node_8304

node_8309

node_16248

node_8063

node_8065

node_8292

node_8302

node_16255

node_7962

node_7965

node_16259

node_7952

node_7954

node_16266

node_16270

node_20804node_20800

node_16162

node_8132

node_11147node_8328 node_8134

node_11151node_8330

node_16166

node_11139 node_11130

node_11134

node_16174

node_11050

node_7983

node_11054node_7985

node_16176

node_11043

node_11046

node_16183node_11518

node_16204

node_16189

node_11167node_8324 node_8116

node_8118

node_11171node_8326 node_8124

node_8126

node_16193

node_11157node_8320 node_8106

node_8108

node_8114

node_11161node_8322

node_16200 node_11058

node_7975

node_11062node_7977

node_11066

node_7979

node_11070node_7981

node_21127

node_21132

node_21142

node_14288node_14286node_9486

node_14293

node_9474

node_9476

node_9479

node_14300

node_7302

node_9560node_7342 node_7300

node_9564node_7340

node_14306

node_9459

node_9461

node_9463

node_14310

node_9546

node_7294

node_9550node_7334

node_14404

node_14409

node_7358

node_9640node_7390 node_7356

node_9644

node_7382

node_7386

node_14413

node_7432

node_9720node_7474

node_9724 node_7424

node_7428 node_7466

node_7470

node_14420

node_9626

node_7350

node_9630node_7370

node_14423

node_9706

node_7412

node_9710node_7454

node_21146

node_14534

node_14537

node_14542

node_9287node_5796

node_5768 node_5760

node_5762

node_5885 node_5875

node_5879

node_14552

node_6226

node_6448

node_6152

node_6156

node_6409

node_6413

node_14560

node_5738

node_9280node_5863

node_14563

node_6128

node_6347

node_14671

node_14673

node_14680

node_6562

node_6628

node_6632

node_6550

node_6554

node_14685

node_6881

node_6893

node_7224

node_7230

node_14694

node_6534

node_6618

node_14697

node_6867

node_7195

node_21153

node_21159 node_21171

node_21176

node_20579

node_20584

node_20592

node_15725

node_15703

node_10671 node_7898

node_10669node_7912

node_15706

node_10633 node_7872

node_10631node_7886

node_15711

node_7892

node_10659node_7906 node_8229

node_10663node_8233

node_15716

node_10621 node_7862

node_7866 node_7883node_7878

node_8221

node_10625node_8225

node_15765

node_10741

node_10744

node_15768

node_10707 node_7926

node_10705node_7932

node_15773

node_10729

node_10734node_10732node_7936

node_15779

node_7922

node_10695node_7928 node_8237

node_10699node_8239

node_15788node_11420

node_20595

node_15586

node_15565

node_7760

node_7777

node_15568

node_7732

node_7746

node_15573

node_7754

node_7771

node_8190

node_8198

node_15578

node_7726

node_7740

node_8178

node_8184

node_15632

node_7832node_10608node_7852

node_15635

node_7792

node_7812

node_15640

node_10601node_8215 node_7842

node_7846

node_8211 node_7822

node_7826

node_15649

node_8202

node_8207

node_7788node_7783

node_7802

node_7806

node_15657

node_15660

node_20601

node_20605 node_20612

node_20618

node_19906

node_7250

node_9413node_7268 node_7290

node_9508node_7330

node_14614node_9666 node_7346

node_9588node_7366

node_19909

node_14216 node_5667

node_5835 node_5929

node_6288

node_14349 node_6485

node_6592 node_6721

node_9363node_7128

node_19914

node_19919

node_19928

node_19933 node_19941

node_19946

node_20109

node_16529

node_16655

node_16659

node_20115

node_14129

node_9744

node_9808 node_16437node_9884

node_20127node_20123

node_16662node_11592

node_11594

node_16666node_11596

node_20167node_20164

node_11576

node_16644

node_8545

node_11582node_8547

node_8537

node_11586node_8539 node_8541

node_11590node_8543

node_20179node_20175

node_16649 node_8528

node_11569node_8531 node_8533

node_8535

node_16653 node_8516

node_8518 node_8522

node_8524

node_19879

node_19887 node_19895

node_19900

node_14205

node_5635

node_5808

node_14208

node_5896

node_6251

node_14343 node_6458

node_6566 node_9360node_6659

Page 42: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive
Page 43: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

PrimitiveObject 0

PrimitiveObject 1

PrimitiveObject 2

Page 44: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

PrimitiveObject 0

PrimitiveObject 1

PrimitiveObject 2

(coherence queues)

Page 45: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Automatically finds coherence paths

Page 46: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

The Scene Hierarchy Generator

Page 47: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

System Memory Bus

System Memory Interface

Scene Hierarchy Generator

Course Grain

SchedulerVertex

Data Master

USC0 USC1Texture Unit

USCn-1 USCnTexture Unit

Unified Shading Cluster Array

World-space Trianglesand associated varyings

Page 48: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Limitations

• Scene is represented by triangles – same as today

• BVH is in a defined format optimised for construction and traversal

• Triangle order must generally follow a spatially coherent flow

• An approximate scene scale estimation is needed

• Geometry shaders are not inline with the ray tracing pipeline

Page 49: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Strengths• Shading cluster workload is no higher than a vertex shader

• Only needs to process geometry that actually moved in world space

• Unique algorithm constrains working set to internal registers only

• Single pass operation: in-line with vertex shader execution

• Handles the “long skinny triangles” problem well

• Streaming writes to external memory

• Losslessly compressed output formats due to build algorithm

• Compact logic

Page 50: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Built on sparse log2 oct-tree scaffolding

LOD = 1 LOD = 2 LOD = 3Level = 1 Level = 2 Level = 3

Page 51: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

ProcessTriangle() AssembleParents()

Leaf VoxelCache

VBExtents

On

evic

tion Tree

VoxelCache

Output Scene Acceleration

Structure

Select LOD

For each triangle

Foreach VBNode in

LOD

Compute Parent Voxel

On VoxelCache HIT

VBNode Pool

Per LOD VBNode linked list

headpointers

GenerateVBNode()

v0

v1

v2

On

evic

tion

Add VBNode

Page 52: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

After some triangles...

LOD16

LOD15

LOD10

LOD14

LeafCount=8TreeCount=0

LeafCount=3TreeCount=0

LeafCount=0TreeCount=0

LeafCount=3TreeCount=0

Leaf Head

Tree Head

Leaf Head

Tree Head

Leaf Head

Tree Head

Leaf Head

Tree Head

Page 53: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Assembling parents...

LOD16

LOD15

LOD10

LOD14

LeafCount=8TreeCount=0

LeafCount=3TreeCount=0

LeafCount=0TreeCount=0

LeafCount=3TreeCount=0

Leaf Head

Tree Head

Leaf Head

Tree Head

Leaf Head

Tree Head

Leaf Head

Tree Head

Tree VoxelCache

On eviction

Flush Leaf Nodes

VBNode Pool

GenerateVBNode()

Add VBNode to linked list of

VBNodes for its LOD

Page 54: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

After parents for one level have been assembled...

LOD16

LOD15

LOD10

LOD14

A0 A1 B0 C0 C1 B1 B2 C2LeafCount=8TreeCount=0

LeafCount=3TreeCount=0

LeafCount=0TreeCount=0

LeafCount=3TreeCount=0

Leaf Head

Tree Head

Leaf Head

Tree Head

Leaf Head

Tree Head

Leaf Head

Tree Head

A B C

Page 55: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Performance Expectations

Page 56: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

System Memory Bus

VertexData Master

PixelData Master

ComputeData Master

Course Grain

Scheduler Core Management

Memory Cache Unit

2D Core (PTLA)

Pixel Co-Processor

Tiling Co-processor

Control and Register BusHost CPU Interface

RayData Master

System Memory Interface

Frame Accumulator

Cache

Scene Hierarchy Generator

Intersection Processor Array Coherency Engine

Ray Tracing Unit

USC0 USC1Texture Unit

USCn-1 USCnTexture Unit

Unified Shading Cluster Array

...

6 clk/tri1-2 clk/ray

64 AABB/ray16 Tri/ray

1-2clk/accum

128 Inst/ray(512flops peak)

Page 57: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Get early access to the programming concepts at:http://tinyurl.com/9ddlsv2

Talk immediately after this on Hybrid rendering

See the demos at Imagination Booth #402 South Hall

Page 58: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

QA

Page 59: Introduction to PowerVR Ray Tracingtwvideo01.ubm-us.net/o1/vault/GDC2014/...•Series6 gives you over 100Gflops for shading on latest mobile devices •OpenGL ES 3.0 support •Massive

Thank you