CS 354Blending, Compositing, Anti-aliasing

Mark KilgardUniversity of TexasFebruary 14, 2012

CS 354 2

Today’s material

In-class quiz Lecture topic: blending, compositing, anti-

aliasing How are colors combined How do we minimize color sampling artifacts

Assignment Reading

Chapter 7, pages 404-420 You should be working on Project #1

Due Tuesday, February 21

CS 354 3

Administrative Homework #2 solution available

http://www.cs.utexas.edu/~mjk/teaching/cs354_s12/hw2answers.pdf

Homework #2 grading Points possible: 39 (100%) Average, excluding zeros: 33 (82%)

Quiz feedback continuing Look on piazza

CS 354 4

My Office Hours

Tuesday, before class Painter (PAI) 5.35 8:45 a.m. to 9:15

Thursday, after class ACE 6.302 11:00 a.m. to 12:00

CS 354 5

Last time, this time

Last lecture, we discussed How do we look at objects How do we represent objects for interactive

rendering This lecture

Object representation Blending colors Compositing images Anti-aliasing images

CS 354 6

Daily Quiz

1. Multiple choice: Which of these vectors is NOT an input vector parameter to the “look at” view transform:

a) frustum vectora) eye vectorb) at vectorc) up vector

2. Multiple choice: The OpenGL convention (assuming only positive saling factors) for eye space has the eye looking down this axis direction:

a) positive Xb) negative Xc) positive Yd) negative Ye) positive Zf) negative Z

3. Multiple choice: The 4x4 matrix generated by the “look at” transform is, in its general form, a:

a) an identity matrixb) an affine matrixc) a projective matrix

On a sheet of paper• Write your EID, name, and date• Write #1, #2, #3, followed by its answer

CS 354 7

Representing Objects Interested in object’s boundary (or manifold) Various approaches

Procedural representations Often fractal

Explicit polygon (triangle) meshes By far, the most popular method

Curved surface patches Often displacement mapped

Implicit representation Blobby, volumetric

Sierpinski gasket

Utah Teapot

Blobby modeling in RenderMan

Quake 2 key frame triangle meshes

Fractaltree

[Philip Winston]

CS 354 8

Focus on Triangle Meshes Easiest approach to representing object boundaries So what is a mesh and how should it be stored?

Simplest view A set of triangles, each with its “own” 3 vertices

Essentially “triangle soup” Yet triangles in meshes share edges by design

Sharing edges implies sharing vertices More sophisticated view

Store single set of unique vertexes in array Then each primitive (triangle) specifies 3 indices into array of vertexes More compact

Vertex data size >> index size Avoids redundant vertex data

Separates “topology” (how the mesh is connected) from its “geometry” (vertex positions and attributes)

Connectivity can be deduced more easily Makes mesh processing algorithms easier Geometry data can change without altering the topology

CS 354 9

Consider a Tetrahedron Simplest closed volume

Consists of 4 triangles and 4 vertices (and 4 edges)

v0

v1

v3

v2 triangle list0: v0,v1,v21: v1,v3,v2 2: v3,v0,v23: v1,v0,v3(x0,y0,z1)

(x1,y1,z1)

(x2,y2,z2)

(x3,y3,z3)

vertex list0: (x0,y0,z0)1: (x1,y1,z1)2: (x2,y2,z2)3: (x3,y3,z3)

topology geometry potentially on-GPU!

CS 354 10

Drawing the Tetrahedron Expanded Immediate mode

glBegin(GL_TRIANGLES); { glVertex3f(x0,y0,z0); glVertex3f(x1,y1,z1); glVertex3f(x2,y2,z2);

glVertex3f(x1,y1,z1); glVertex3f(x3,y3,z3); glVertex3f(x2,y2,z2);

glVertex3f(x3,y3,z3); glVertex3f(x0,y0,z0); glVertex3f(x2,y2,z2);

glVertex3f(x1,y1,z1); glVertex3f(x0,y0,z0); glVertex3f(x3,y3,z3);} glEnd();

Indexed vertexes

glBegin(GL_TRIANGLES); { for (int i=0; i<4; i++) { glVertex3fv(vertex[triangle[i].v0]); glVertex3fv(vertex[triangle[i].v1]); glVertex3fv(vertex[triangle[i].v2]); }} glEnd();

Client-memory Vertex arrays

GLuint ndxs[12] = { 0,1,2, 1,3,2, 3,0,2, 1,0,3 };glEnableClientState( GL_VERTE

X_ARRAY);glVertexPointer (3, GL_FLOAT,

3*sizeof(GLfloat), xyz); glDrawElements(GL_TRIANGLES, 12, GL_UNSIGNED_INT, ndxs);

CS 354 11

Benefits of Vertex Array Approach

Unique vertices are stored once Saves memory

On CPU, on disk, and on GPU Matches OpenGL vertex array model of

operation And this matches the efficient GPU mode of

operation The GPU can “cache” post-transformed vertex results by

vertex index Saves retransformation and redundant vertex fetching

Direct3D has the same model Allows vertex data to be stored on-GPU for even

faster vertex processing OpenGL supported vertex buffer objects (VBOs) for this

CS 354 12

More Information

See “Modern OpenGL Usage: Using Vertex Buffer Objects Well” http://www.slideshare.net/Mark_Kilgard/using-vertex-bufferobjectswell

CS 354 13

A Simplified Graphics PipelineApplication

Vertex batching & assembly

Triangle assembly

Triangle clipping

Triangle rasterization

Fragment shading

Depth testing

Color update/blending

Application-OpenGL API boundary

Framebuffer

NDC to window space

Depth buffer

Re-examineframebuffer color

update…

CS 354 14

A few more steps expandedApplication

Vertex batching & assembly

Lighting

View frustum clipping

Triangle rasterization

Fragment shading

Depth testing

Color update/blending

Application-OpenGL API boundary

Framebuffer

NDC to window space

Depth buffer

Vertex transformation

User defined clipping

Back face culling

Perspective divide

Triangle assemblyTexture coordinate generation

CS 354 15

Blending

Simple operation Two inputs

Color value currently in framebuffer for pixel Shaded color value of fragment rasterized at pixel

One output A new “blended” color

pixel color

fragment color

blend operation

CS 354 16

Blending Enabled vs. Disabled

pixel color

fragment color

blend operation

pixel color

fragment color

glDisable(GL_BLEND) glEnable(GL_BLEND)

CS 354 17

RGBA: Red, Green, Blue, Alpha

Four-component colors Red, green, blue

Measures of color component intensity From 0% to 100% Often stored as 8-bit unsigned values

Alpha—measure of opacity Also 0% (fully transparent) to 100% (fully opaque)

CS 354 18

Meaning of Alpha Translucency = 100% – Opacity

Fully opaque surfaces permit no light to pass through surface

Translucent surfaces permit some light to pass through surface

Best though of in probabilistic terms Implies uncertainty about geometry of occlusion at

the sub-pixel level

CS 354 19

Why blending?

compositing window systems

volumetric effects; explosions

medical imaging

compositingcomplexart work

CS 354 20

Conventional Blend Operation

sourcecolor

destinationfactor

destinationcolor

sourcefactor

××

+

clamp [0,1]

pixel color

fragment color

CS 354 21

Conventional Blend Operation

sourcecolor

destinationfactor

destinationcolor

sourcefactor

××

+

clamp [0,1]

pixel color

fragment color

×××

+++

×××

clamp [0,1]clamp [0,1]clamp [0,1]

modulate, add, and clamp operations are vector on RGBA components

CS 354 22

Conventional Blend Operation

sourcecolor

destinationfactor

destinationcolor

sourcefactor

××

+

clamp [0,1]

pixel color

fragment color glBlendFunc(srcFunc, dstFunc)

CS 354 23

Blend Function ParametersParameter (fr, fg, fb, fa)

GL_ZERO (0,0,0,0)

GL_ONE (1,1,1,1)

GL_SRC_COLOR (Rs,Gs,Bs,As)

GL_ONE_MINUS_SRC_COLOR (1-Rs,1-Gs,1-Bs,1-As)

GL_DST_COLOR (Rd,Gd,Bd,Ad)

GL_ONE_MINUS_DST_COLOR (1-Rd,1-Gd,1-Bd,1-Ad)

GL_SRC_ALPHA (As,As,As,As)

GL_ONE_MINUS_SRC_ALPHA (1-As,1-As,1-As,1-As)

GL_DST_ALPHA (Ad,Ad,Ad,Ad)

GL_ONE_MINUS_DST_ALPHA (1-Ad,1-Ad,1-Ad,1-Ad)

GL_CONSTANT_COLOR (Rc,Gc,Bc,Ac)

GL_ONE_MINUS_CONSTANT_COLOR (1-Rc,1-Gc,1-Bc,1-Ac)

GL_CONSTANT_ALPHA (Ac,Ac,Ac,Ac)

GL_ONE_MINUS_CONSTANT_ALPHA (1-Ac,1-Ac,1-Ac,1-Ac)

GL_SRC_ALPHA_SATURATE (s,s,s,s)where s = min(As,1-Ad)

CS 354 24

glBlendFunc Example: Over

glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA);

Meaning srcColor + (1 – srcAlpha)×dstColor So called “over” operation

Source color blended “over” destination color Render layers bottommost-to-topmost

1 23

CS 354 25

glBlendFunc Example: Under

glBlendFunc(GL_ONE_MINUS_DST_ALPHA, GL_ONE);

Meaning (1 –dstAlpha)×srcColor + dstColor So called “under” operation

Source color blended “under” destination color Render layers topmost-to-bottommost

3 21

CS 354 26

Pre-multiplied Alpha

Opacity should be multiplied in color components Essentially (R×α,G×α,B×α,α)

Utility of pre-multiplied alpha isn’t obvious Non-pre-multiplied alpha says the RGB components

don’t include opacity Essentially (R,G,B,α)

Sometimes called “straight” color But wrong because such “straight” colors don’t combine

properly mathematically

CS 354 27

Hardware View of Blending Each blend operation means

Read-Modify-Write (RMW) operation More expensive than just a read or write

Implies memory bus must be “turned around”

GPUs perform blending in special Raster Operation (ROP) units Good example of fixed-function graphics hardware

Strategies for performance Group RMW operations for multiple pixels Organize framebuffer for 2D memory locality Data-dependent discards and RMW-to-W conversions Color compression

CS 354 28

Why not do Blending in the Fragment Shader?

Blending is fairly simple math Programmable fragment shading is much more

general So why not do the blending operations in the shader?

Good reason The Read-Modify-Write of a blend operation requires

an “interlock” Pixels must be blended in primitive order If shader does blend, means shader can only be processing

one fragment for any given pixel at a time If shader can “see” the pixel color of a fragment, all the prior

fragments bound for the pixel must be completed in order to start shading the pixel again

This “interlock” would limit shading performance

CS 354 29

Sophisticated Blending Since OpenGL 1.0, blending functionality has been

embellished Embellishments

Constant blend color glBlendColor

Blend equation glBlendEquation—min, max, subtract, reverse subtract

Separate RGB and alpha blend functions and equations glBlendEquationSeparate, glBlendFuncSeparate

Distinct blending controls for multiple color buffers Also known as render targets in Direct3D

Multi-sampling for anti-aliasing Implies per-color sample blending

Floating-point (single- and half-precision) blending

CS 354 30

Newer Blend Commands glBlendColor(red,green,blue,alpha)

Accessed by the GL_CONSTANT_COLOR, etc. blend functions glBlendEquation(func)

func is on of GL_FUNC_ADD, GL_MIN, GL_MAX, GL_FUNC_SUBTRACT, GL_FUNC_REVERSE_SUBTRACT

glBlendFuncSeparate Example of “over” for straight RGBA values:

glBlendFuncSeparate( GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_SRC_ALPHA)

Allows straight RGBA to composite correctly glBlendEquationSeparate

Different blend equation for RGB versus alpha

CS 354 31

Blend Color for Factors

sourcecolor

destinationfactor

destinationcolor

sourcefactor

××

+

clamp [0,1]

pixel color

fragment colorglBlendColor(r,g,b,a)

blendcolor

CS 354 32

Min/Max Blend Operation

sourcecolor

destinationcolor

min or max

clamp [0,1]

pixel color

fragment color glBlendEquation(GL_MIN)

glBlendEquation(GL_MAX)

CS 354 33

Consider Example Consider straight colors

50% of (1,0,0,1) and 50% of 100% red

50% of (0,0,1,0.2) 50% of 20% blue

Result of weighted average of components (0.5,0,0.5,0.6)

60%-opaque magenta? Non-sensible when much less blue than red

Proper result Pre-multiplied colors are (1,0,0,1) and (0,0,0.2,0.2) Now weighted average components:

(0.5,0.1,0.6) or 60%-opaque maroon red? Sensible that result is mostly red

CS 354 34

“Over” BlendingNot Commutative

Order of blending matters! blend(blend(A,B), C) ≠ blend(blend(C,A), B)

Also blend(A,B) ≠ blend(B,A) Blending is not commutative

Can’t re-arrange blend operations (Similar to matrix composition)

Pre-multiplied alpha blending for over and under is associative blend(blend(A,B), C) = blend(A, blend(B,C))

Reverse of over blending is under So back-to-front = front-to-back But requires framebuffer maintain a destination alpha channel

CS 354 35

Getting Blending Right

Blending operations must be properly ordered

How to do this? Sort your objects Sort you triangles

Neither of above is always possible Sort your fragments

Depth peeling A-buffer schemes

CS 354 36

Properly Ordered Compositingvs. Incorrectly Ordered

CS 354 37

Properly Ordered Compositingvs. Incorrectly Ordered

CS 354 38

Compositing

Blending operates on pixels Compositing operates on images

Composite image A & image B

CS 354 39

Intra-pixel Regions for Compositing

A ∩ B

A ∩ ~B

~A ∩ B

~A ∩ ~B Source: SVG Compositing Specification

CS 354 40

Compositing Digital Images

Classic 1984 SIGGRAPH paper introduces compositing operators Porter and Duff

Porter-Duff Composite Operators Rca = f(Ac,Bc)×Aa×Ba + Y×Ac×Aa×(1-Ba) + Z×Bc×(1-Aa)×Ba Ra = X×Aa×Ba + Y×Aa×(1-Ba) + Z×(1-Aa)×Ba

CS 354 41

Porter-DuffComposite Operators

CS 354 42

Porter & Duff ModesOperation f(Ac,Bc) X Y ZClear 0 0 0 0Src Ac 1 1 0Dst Bc 1 0 1

Src-Over Ac 1 1 1Dst-Over Bc 1 1 1Src-In Ac 1 0 0Dst-In Bc 0 1 0Src-out 0 0 1 0Dst-out 0 0 0 1Src-atop Ac 1 0 1Dst-atop Bc 1 1 0Xor 0 0 1 1

Porter & Duff blend modes

CS 354 43

Porter & Duff Modes ExpandedOperation f(Ac,Bc) X Y Z Blend modeClear 0 0 0 0 0Src Ac 1 1 0 AcaDst Bc 1 0 1 BcaSrc-Over Ac 1 1 1 Aca+(1-Aa)×BcaDst-Over Bc 1 1 1 Bca+(1-Ba)×AcaSrc-In Ac 1 0 0 Aca×BaDst-In Bc 0 1 0 Bca×AaSrc-out 0 0 1 0 (1-Ba)×AcaDst-out 0 0 0 1 (1-Aa)×BcaSrc-atop Ac 1 0 1 Aca×Ba+(1-Aa)×BcaDst-atop Bc 1 1 0 (1-Ba)×Aca+Aa×BcaXor 0 0 1 1 Aca×(1-Ba)+(1-Aa)×Bca

Uncorrelated blend mode expansion of Porter & Duff blend modes

CS 354 44

Porter & Duff for glBlendFuncOperation Blend mode srcFactor dstFactorClear 0 GL_ZERO GL_ZERO

Src Aca GL_ONE GL_ZERO

Dst Bca GL_ZERO GL_ONE

Src-Over Aca+(1-Aa)×Bca GL_ONE GL_ONE_MINUS_SRC_ALPHA

Dst-Over Bca+(1-Ba)×Aca GL_ONE_MINUS_DST_ALPHA

GL_ONE

Src-In Aca×Ba GL_DST_ALPHA GL_ZERO

Dst-In Bca×Aa GL_ZERO GL_SRC_ALPHA

Src-out (1-Ba)×Aca GL_ONE_MINUS_DST_ALPHA

GL_ZERO

Dst-out (1-Aa)×Bca GL_ZERO GL_ONE_MINUS_SRC_ALPHA

Src-atop Aca×Ba+(1-Aa)×Bca GL_DST_ALPHA GL_ONE_MINUS_SRC_ALPHA

Dst-atop (1-Ba)×Aca+Aa×Bca GL_ONE_MINUS_DST_ALPHA

GL_DST_ALPHA

Xor Aca×(1-Ba)+(1-Aa)×Bca GL_ONE_MINS_DST_ALPHA GL_ONE_MINUS_SRC_ALPHA

CS 354 45

Hardware Blending supports all Porter-Duff Blend Modes

Using prior slide’s table Your OpenGL (or Direct3D) program can implement any of

Porter-Duff blend modes Examples

Src-Over glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA)

Dst-In glBlendFuc(GL_ZERO, GL_SRC_ALPHA)

Dst-Atop glBlendFunc(GL_ONE_MINUS_DST_ALPHA, GL_DST_ALPHA)

Conclusion: GPU hardware “blend functions” can configure all the sound Porter-Duff compositing algebra blend modes Compositing algebra theory “maps” well to GPU functionality Assumption: using pre-multiplied alpha colors

CS 354 46

Additional Blend Modes

Additional blend modes Since Porter-Duff’s composite operators,

Adobe introduced further artistic blend modes Part of PhotoShop, Illustrator, PDF, Flash,

and other standards Part of the vocabulary of digital artists now

Examples ColorDodge, HardLight, Darken, etc.

Define with alternate f(Ac,Bc) function

CS 354 47

Aliased

Jaggedartifacts

CS 354 48

Multi-sample8x

Smootherappearance

CS 354 49

Multi-sample Coverage Positions

4x jittered1x(aliased)

8x jittered

4x orthogonal

CS 354 50

Next Lecture

Color representation What ways can quantitatively represent color? As usual, expect a short quiz on today’s lecture

Assignments Reading

Chapter 7, pages 404-420 Work on Project #1

Building a 3D object model loader Due Tuesday, February 21