2D Texture Synthesis

Instructor: Yizhou Yu

Texture synthesis

Goal: increase texture resolution yet keep local texture variation

Synthesis by global statistics Idea: Obtain statistics of input image,

match with output image

Histograms: normalized graph of intensity frequencies

Synthesis by global statistics

Consider extreme case:

Histogram matching can generate two images:

Using higher-order global stats Histogram matching does not take into

account spatial information. Need additional info:

For every pixel, examine local neighborhood Gives local but not global features (e.g. – veins)

Use higher order stats: Correlation

average of the product of the intensities =

tells how interdependent pixel values are

1[ ( ) ( )] ( ) ( )

x

E I x I x I x I xn

General procedure 1.) Define error metric:

What do you want to match?

2.) Match statistics Histogram matching:

Get the density value of a pixel in the output image. Map this density to a pixel intensity in the input image Overwrite the pixel in the output image with the mapped pixel intensity in the input image

Pyramid-based Texture Analysis/Synthesis

Paper by David J. Heeger and James R. Bergen from SIGGRAPH 1995

http://www.cns.nyu.edu/~david/ftp/reprints/heeger-siggraph95.pdf

Pyramid-based Synthesis Downsample image several times and

keep track of only differences in a feature image

To reconstruct, upsample small images and add differences

Pixels in feature images are close to zero Provides multiple scales with each feature

image representing different feature sizes

Laplacian Pyramid

1-D filter: (1/16) (1 4 6 4 1) To generalize to 2D, use tensor

product

Normalize by dividing by 256

Upsampling To upsample, copy pixels to every

other pixel in larger image

Fill the rest with zeroes, run low pass filter to fill in values

Oriented Filters

Gaussian filters are symmetric, thus edges and contours are not detected

Use multiple oriented filters to catch non-symmetric features

Left top series: Oriented filters Right image: Texture

Left bottom series: Filtered textures

Pyramids with Oriented Filters

Each oriented filter creates its own feature image

Thus, for each downsampled image, keep one image for each oriented filter

Each oriented filter captures one orientation of lines

Matching Image Features

Input parameters: noise: initial noisy texture texture: texture to be matched

Output texture is stored in noise

Matching Image Features Helper functions:

MatchHistogram(noise, texture) Matches histogram, using method described last

time MakePyramid(texture)

Create pyramid images (base and feature images) CollapsePyramid(pyramid)

Constructs high resolution image from base and feature images

Matching Image FeaturesMatchTexture(noise, texture){

MatchHistogram(noise, texture) // first-order matchinganalysis_pyr = MakePyramid(texture) // create pyramid from input texturefor several iterations

synthesis_pyr = MakePyramid(noise) // create pyramid from noise

for each feature image, fi of analysis_pyrfor each feature image, fj of synthesis_pyr of same

orientationMatchHistogram(fi, fj)

end forend fornoise = CollapsePyramid(synthesis_pyr)MatchHistogram(noise, texture)

end for}

Matching Image Features MatchTexture matches histograms of

feature images, not pixel values, thus providing a much better matching

Feature images already consider local neighborhoods, resulting in better approximations

Good for randomized textures Textures with large scale features or

thin and long features not matched well e.g. stripes in wood, lines in coral

Texture Synthesis Using Local Neighborhoods Main goal is to keep local spatial

coherence, but not global stats. Randomized method

Pick pixel and copy it along with its neighborhood to random parts in synthesized image

If two neighborhoods overlap, just blend This can result in features getting cut

off if larger than local neighborhood

Neighborhood-Based Texture Synthesis

Patch-Based Patch-based sampling achieves real-time speed.

[Liang et. al. 2001] Image quilting: high-quality results and simple

implementation [Efros & Freeman 2001] Graph cut provides a powerful and refinable

scheme. [Kwantra et. al. 2003] Pixel-Based

[Efros & Leung 99], [Wei & Levoy 2000], [Ashikhmin 2001], [Hertzmann et. al. 2001], [Zhang et. al. 2003]

Pixel-wise Synthesis Grow pixel by pixel Start from an existing patch of the

texture (as opposed to noise texture like Pyramid-based Synthesis)

Look for regions in input texture most similar to current region in new texture

Copy pixels next to best-match region to expand new texture

Following Raster Order Blue region: already set by algorithm Green region: compare this region to input image Yellow region: closest match Red region: replace these pixels with magenta

region to maintain local integrity

Hierarchical Synthesis

Build a multi-resolution pyramid for the example texture

Generate a synthesized pyramid for the output texture At each level, follow the raster order

Randomizing Synthesis Instead of picking pixels from closest

matching region, use threshold to pick a few candidate matches

Randomly pick one of the candidates Refer to “Texture Synthesis by Non-

parametric Sampling” http://www.cs.berkeley.edu/~efros/

research/synthesis.html

Results

Patch-Based Synthesis

Search in a sample texture for neighborhoods most similar to a context region

Merge a patch with the partially synthesized output texture

The Seam Problem

Feature discontinuities may appear in patch-based synthesis.

The Second Reason Didn’t find the smoothest transition

between the incoming patch and context region.

Solutions: use dynamic programming [Efros and Freeman 2001] or graph cut [Kwatra et al. 2003] to find an optimal cut.

Original Warped

The First Reason Rigid template

matching (SSD) often employed in the first step does NOT consider geometric similarity

Original Warped

Feature Map Guided Texture Synthesis

Basic Steps [Wu and Yu 2004]

Maintain an input and output feature map Match and align curvilinear features Integrate feature maps into template-

based texture synthesis

Comparisons

Sample Feature Map Graphcut Quilting Texton Mask

Acceleration Schemes Fourier Transform Acceleration techniques for nearest-neighbor

search Tree-structured vector quantization Kd-trees

Minimizing the candidate set Coherent synthesis, [Ashikhmin 2001]

Precomputing candidate set K-coherent search, [Tong et al. 2002] Jump Map, [Zelinka and Garland 2002]

GPU with parallel synthesis [Lefebvre and Hoppe 2005]

Original Warped