CS380: Introduction to Computer Graphics Reconstruction ...
Transcript of CS380: Introduction to Computer Graphics Reconstruction ...
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
CS380: Introduction to Computer GraphicsReconstruction & Resampling
Chapter 17 & 18
Min H. KimKAIST School of Computing
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
SUMMARYSampling (continuous à discrete)
2
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Aliasing• Scene made up of black and white triangles: jaggies
at boundaries– Jaggies will crawl during motion
• If triangles are small enough then we get random values or weird patterns– Will flicker during motion
• The heart of the problem: too much information in one pixel
3
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Anti-aliasing• We can also model this as an optimization problem.• These approaches lead to:
• where is some function that tells us how strongly the continuous image value at
should influence the pixel value
4
I[i][j]← I(x , y)Ω∫∫ Fi , j(x , y)dxdy
Fi , j(x , y)
[x , y]t i,j
I[i][j]
I(xw , yw )
I(x , y)←
ΩFi , jxw = i , yw = j
Fi , j(x , y)
I(x , y)
Vertical section view
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
• Even that integral is not really reasonable to compute• Instead, it is approximated by some sum of the form:
where k indexes some set of locations called the sample locations, so-called over-sampling.
• The renderer first produces a “high resolution” colorand z-buffer “image”,– where we will use the term sample to refer to each of
these high resolution pixels.
Over-sampling
5
I[i][j]← 1
nI(xk , yk )
k=1
n
∑
(xk , yk )
a pixel
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Premultiplied Alpha definition• More specifically, let be a continuous
image, and let be a binary valued coverage function over the continuousdomain, with a value of 1 at any point where the image is “occupied” and 0 where it is not.
• Let us store in our discrete image the values:
6
I(x, y)C(x, y)
(x, y)
I[i][j]← I(x , y)
Ωi , j∫∫ C(x , y)dxdy
α[i][j]← C(x , y)dxdy
Ωi , j∫∫
x ⊂ xw = i , y ⊂ yw = j
I[i][j] I(xw , yw )
I(x , y)←
Ωi , j Ωi , j
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Premultiplied alpha: Over operation• To compose , we compute
the composite image colors, , using
• That is, the amount of observed background color at a pixel is proportional to the transparency of the foreground layer at that pixel.
• Likewise, alpha (coverage) for the composite image can be computed as:
7
If[i][j] over Ib[i][j]Ic[i][j]
Ic[i][j]← If [i][j]+Ib[i][j](1-α f [i][j])
αc[i][j]← α f [i][j]+α b[i][j](1-α f [i][j])
Including alpha
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Alpha blending error• The difference between two
• Therefore the error is the difference between the integral of a product and a product of integrals.
• It is the amount of covariance between the distribution of foreground coverage in some pixel and the distribution of the background data within that pixel.
8
error= Ib(x , y)Cb(x , y)C f (x , y)dxdyΩi , j∫∫ −
Ib(x , y)Cb(x , y)dxdy× C f (x , y)dxdyΩi , j∫∫Ωi , j
∫∫
cov(X ,Y )=E[XY ]−E[X ]E[Y ]
Real alpha blending
Ideal continuous composition
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
RECONSTRUCTION(DISCRETE à CONTINUOUS)
Chapter 17
9
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Reconstruction• Given a discrete image I[i][j], how do we create
a continuous image I(x,y)?• It is the key problem for resizing images and
texture mapping.– How to get texture colors that fall in between texels.
• This process is called reconstruction.
10
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Constant reconstruction• A real valued image coordinate is assumed to
have the color of the closest discrete pixel. This method can be described by the following pseudo-code:
• The (int) typecast rounds a number p (pixel position) to the nearest integer not larger than p.
11
color constantReconstruction(float x, float y, color image[][]){int i = (int) (x + .5);int j = (int) (y + .5);return image[i][j];
}
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Constant reconstruction• The resulting continuous image is made up of
little squares of constant color.• Each pixel has an influence region of 1-by-1
12
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Bilinear properties• At integer coordinates, we have I(x,y)=I[i][j]; the
reconstructed continuous image I agrees with the discrete image I.
• In between integer coordinates, the color values are blended continuously.
• Each pixel in the discrete image influences, to a varying degree, each point within a 2-by-2 square region of the continuous image.
• The horizontal/vertical ordering is irrelevant.• Color over a square is bilinear function of (x,y).
13
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Bilinear function• 1 by 1 square with coordinates
for some fixed .
• where are the fracx and fracy above.
14
j < y < j +1i < x < i +1
i and j
I(i+ x f , j+ y f )←(1− y f ) (1− x f )I[i][j]+(x f )I[i+1][j]{ }+( y f ) (1− x f )I[i][j+1]+(x f )I[i+1][j+1]{ } ,
x f and yf
x f
y f
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Bilinear reconstruction• Can create a smoother looking reconstruction using
bilinear interpolation.• Bilinear interpolation is obtained by applying linear
interpolation in both the horizontal and verticaldirections.
15
color bilinearReconstruction(float x, float y, color image[][]){int intx = (int) x;int inty = (int) y;float fracx = x - intx;float fracy = y - inty;
color colorx1 = (1-fracx) * image[intx][inty] +(fracx) * image[intx+1][inty];
color colorx2 = (1-fracx) * image[intx][inty+1] +(fracx) * image[intx+1][inty+1];
color colorxy = (1-fracy) * colorx1 +(fracy) * colorx2;
return(colorxy);}
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Bilinear function• Rearranging the terms,
• This function has terms that are constant, linear, and bilinear terms in the variables
16
I(i+ x f , j+ y f )← I[i][j]+ −I[i][j]+I[i+1][j]( )x f + −I[i][j]+I[i][j+1]( ) y f + I[i][j]-I[i][j+1]-I[i+1][j]+I[i+1][j+1]( )x f y f
(x f , yf )
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Bilinear basis function• Rearrange the bilinear function to obtain:
• For a fixed position , the color of the continuous reconstruction is linear in the discrete pixel values of I:
17
I(i+ x f , j+ y f )← 1− x f − y f + x f y f( )I[i][j]+ x f − x f y f( )I[i+1][j]+ y f − x f y f( )I[i][j+1]+ x f y f( )I[i+1][j+1]
(x f , yf )
I(x , y)← Bi , j(x , y)I[i][j]
i , j∑
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Bilinear basis function• These B are called basis functions (tent
functions)• They describe how much pixel i, j influences the
continuous image at .• In 1D, we can define a univariate hat function
.
18
[x, y]t
Hi (x)
Hi(x)=x − i+1fori−1< x < i−x+ i+1fori < x < i+1
0else
⎧
⎨⎪
⎩⎪
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
• In 2D (bilinear function), let be a bivariate function:
• This is called a tent function
Bilinear basis function
19
Ti, j (x, y)
Ti , j(x , y)=Hi(x)Hj( y).
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Bilinear basis function• In constant reconstruction, is a box
function that is zero everywhere except for the unit square surrounding the coordinates (i,j), where it has constant value 1.
20
I[i][j]
I(xw , yw )
I(x , y)←Ω
Fi , j
xw = i , yw = j
Bi, j (x, y)
I[i][j]← I(x , y)Ω∫∫ Fi , j(x , y)dxdy
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
RESAMPLING(RECONSTRUCTION+SAMPLING,DISCRETEàCONTINUOUSàDISCRETE)
Chapter 18
21
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Resampling• Let’s revisit texture mapping• We start with a discrete image and end with a
discrete image.• The mapping technically involves both a
reconstruction and sampling stage.• In this context, we will explain the technique of
mip mapping used for anti-aliased texture mapping.
22
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Resampling equation• Suppose we start with a texture image (discrete)
T[k][l] and apply some 2D warp to this image to obtain an output image I[i][j].
23
T[k][l]← T(x , y)Ω∫∫ Fk ,l(x , y)dxdy
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Resampling equation• Reconstruct a continuous texture using
a set of basis functions .
• Apply the geometric wrap (at the view point) to the continuous image.
24
T(xt , yt )Bk ,l(xt , yt )
texturecoordinates(asVV): v
wn
and 1wn
T(xt , yt )← Bk ,l(xt , yt )T[k][l]
k ,l∑
T '(xw , yw )←T(M(xw , yw ))(xt , yt )=M(xw , yw )
M (Ωi, j )
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Resampling equation• Integrate against a set of filters (e.g.,
a box filter) to obtain the discrete output image.
25
Fk ,l(xw , yw )
I[i][j]← T '(xw , yw )Ω∫∫ Fi , j(xw , yw )dxwdywM (Ωi, j )
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Resampling equation• Let the geometric transform be described by a warp
mapping function , which maps from continuous window to texture coordinates.
• We obtain:
26
(xt , yt )=M(xw , yw )
I[i][j]← Fi, j (xw , yw )
Ω∫∫ Bk ,l M (xw , yw )( )k ,l∑ T[k][l]⎛
⎝⎜⎜
⎞
⎠⎟⎟dxwdyw
= T[k][l] Fi, j (xw , yw )
Ω∫∫ Bk ,l M (xw , yw )( )( )dxwdyw( )k ,l∑
I(x , y)← Bi , j(x , y)I[i][j]
i , j∑
(we could obtain an output pixel as a linear combination of the input texture pixels.)
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Resampling equation
• where is the inverse mapping
is the Jacobian matrix of N (changes of area sizes)(continuity of composition of F and N)
27
I[i][j]← det(D
M −1 ) Fi, j M −1(xt , yt )( )( )M (Ωi , j )∫∫ Bk ,lk ,l∑ (xt , yt )T[k][l]⎛
⎝⎜⎜
⎞
⎠⎟⎟dxtdyt
DN
ʹF =F !N
N =M−1
= det(DN ) ʹFi, j (xt , yt )( )M (Ωi , j )∫∫ Bk ,l
k ,l∑ (xt , yt )T[k][l]⎛
⎝⎜⎜
⎞
⎠⎟⎟dxtdyt
1( , ) ( , )w w t tx y M x y-=
• Instead, we can rewrite the integration over the texture domain, instead of the window domain.
I[i][j]← Fi, j (xw , yw )
Ω∫∫ Bk ,l M (xw , yw )( )k ,l∑ T[k][l]⎛
⎝⎜⎜
⎞
⎠⎟⎟dxwdyw
(xt , yt )=M(xw , yw )
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Resampling equation
• In summary, when F is a box filter
• where is how much texel k,l blends to obtain the value at continuous texture location p, from reconstruction step.
28
= det(DN )
M (Ωi , j )∫∫ Bk ,lk ,l∑ (xt , yt )T[k][l]⎛
⎝⎜⎜
⎞
⎠⎟⎟dxtdyt
I[i][j]← Bk ,l M (xw , yw )( )
k ,l∑ T[k][l]⎛
⎝⎜⎜
⎞
⎠⎟⎟dxw dywΩ∫∫
Bk ,l (p)
(xt , yt )=M(xw , yw )
I[i][j]← Fi, j (xw , yw )
Ω∫∫ Bk ,l M (xw , yw )( )k ,l∑ T[k][l]⎛
⎝⎜⎜
⎞
⎠⎟⎟dxwdyw
= det(DN ) ʹFi, j (xt , yt )( )M (Ωi , j )∫∫ Bk ,l
k ,l∑ (xt , yt )T[k][l]⎛
⎝⎜⎜
⎞
⎠⎟⎟dxtdyt
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Texture mapping cases• That is, we need to integrate over the region
on the texture domain, and blend that data together.
• If M is enlarging the texture, then has a very narrow footprint over
• Conversely, if our transformation M effectively shrinks the texture, then has a large footprint over
• During texture mapping, M can also do funnier things, like shrink in one direction only.
29
M(Ωi , j )
M(Ωi , j )T(xt , yt )
M (Ωi, j ) T (xt , yt )
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Enlarging texture• In the case that we are enlarging the texture
(narrow footprint in windows), the filteringcomponent has minimal impact on the output.
• In particular, the footprint of may be smaller than a pixel unit in texture space, and thus there is not much detail that needs blurring/averaging.
• As such, the integration step can be dropped, and the resampling can be implemented as
30
M (Ωi, j )
I[i][j]← Bk ,l(xt , yt )
k ,l∑ T[k][l]⇐ det(DN )M(Ωi , j )
∫∫ Bk ,l(xt , yt )k ,l∑ T[k][l]⎛
⎝⎜⎜
⎞
⎠⎟⎟dxtdyt
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Enlarging texture• We tell OpenGL to do this using the call
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR).
• For a single texture lookup in a fragment shader, the hardware needs to fetch 4 texture pixels and blend them appropriately.
31
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Shrinking texture• In the case that a texture is getting shrunk
down, then, to avoid aliasing, the filtercomponent should not be ignored.
• Unfortunately, there may be numerous texture pixels under the footprint of , and we may not be able to do our texture lookup in constant time.
32
M (Ωi, j )
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
MIP mapping• An acronym of the Latin phrase, multum in
parvo (much in little).• Mipmaps are precalculated scaled versions of an
original texture.
33
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
MIP mapping• Observe the pattern of each mipmap as it decreases
in size; it is half the dimension of the previous one. • This pattern is repeated until the last mipmap’s
dimension is 1×1. • Interpolate multi-resolution textures via trilinear
interpolation
34
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
MIP mapping in detail• In mip mapping, one starts with an original
texture and then creates a series of lower and lower resolution (blurrier) texture .
• Each successive texture is twice as blurry. And because they have successively less detail, they can be represented with ½ the number of pixels in both the horizontal and vertical directions.
35
T 0
T i
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
MIP mapping in detail• This collection, called a mip map, is built before
the any triangle rendering is done.• Thus, the simplest way to construct a mip map is
to average two-by-two pixel blocks to produce each pixel in a lower resolution image.
• During texture mapping, for each texture coordinate , the hardware estimates how much shrinking is going on.– How big is the pixel footprint on the geometry.
36
(xt , yt )
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
MIP mapping in detail• This shrinking factor is then used to select from an
appropriate resolution texture from the mipmap. Since we pick a suitably low resolution texture, additional filtering is not needed, and again, we can just use reconstruction.
• To avoid spatial or temporal discontinuities where/where the texture mip map switches between levels, we can so-called trilinearinterpolation. We use bilinear interpolation to reconstruct one color from and another reconstruction from . These two colors are then linearly interpolated. This third interpolation factor is based on how close we are to choosing level i or i+1
37
T i
T i
T i+1
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
MIP mapping in detail• Mip mapping with trilinear interpolation is
specified with the callglTexParameteri(GL TEXTURE 2D, GL TEXTURE MIN FILTER,GL LINEAR MIPMAP LINEAR)
• Trilinear interpolation requires OpenGL to fetch 8 texture pixels and blend them appropriately for every requested texture access.
38
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Trilinear interpolation• Trilinear interpolation (3D):
39
p(x, y, z) = c0 + c1Δx + c2Δy + c3Δz + c4ΔxΔy + c5ΔxΔzc6ΔyΔz + c7ΔxΔyΔz
p0 + [0,1,n,n +1,n2,
n2 +1,n2 + n,n2 + n +1]The dimension of CLUT is (2n+1)x(2n+1)x(2n+1)
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Trilinear interpolation• where
40
c1 = (p100 − p000 ) / (x1 − x0 )c2 = (p010 − p000 ) / (y1 − y0 )c3 = (p001 − p000 ) / (z1 − z0 )c4 = (p110 − p010 − p100 + p000 ) / [(x1 − x0 )(y1 − y0 )]c5 = (p101 − p001 − p100 + p000 ) / [(x1 − x0 )(z1 − z0 )]c6 = (p011 − p001 − p010 + p000 ) / [(y1 − y0 )(z1 − z0 )]c7 = (p111 − p011 − p101 − p110 + p100 + p001 + p010 − p000 ) /
[(x1 − x0 )(y1 − y0 )(z1 − z0 )].
Δx = x − x0Δy = y − y0Δz = z − z0c0 = p000
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Properties• It is easy to see that mip
mapping does not do the exactly correct computation.
• First of all, each lower resolution image in the mipmap is obtained by isotropic shrinking, equally in every direction. But, during texture mapping, some region of texture space may get shrunk in only one direction.
41
No mipmapping
Mipmapping
Anisotropic mipmapping
Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012
Properties
42
M (Ωi, j )
• Even for isotropic shrinking, the data in the low resolution image only represents a very specific, dyadic, pattern of pixel averages from the original image.
• Filtering can be better approximated at the expense of more fetches from various levels of the mip map to approximately cover the area
on the texture.• This approach is often called ansiotropic filtering
and can be abled in an API or using the driver control panel.