Applied Bayesian Nonparametrics 5. Spatial Models via...

48
Applied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs Tutorial at CVPR 2012 Erik Sudderth Brown University NIPS 2008: E. Sudderth & M. Jordan, Shared Segmentation of Natural Scenes using Dependent Pitman-Yor Processes. CVPR 2012: S. Ghosh & E. Sudderth, Nonparametric Learning for Layered Segmentation of Natural Images.

Transcript of Applied Bayesian Nonparametrics 5. Spatial Models via...

Page 1: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Applied Bayesian Nonparametrics

5. Spatial Models via Gaussian Processes, not MRFs

Tutorial at CVPR 2012Erik Sudderth Brown University

NIPS 2008: E. Sudderth & M. Jordan, Shared Segmentation of Natural Scenes using Dependent Pitman-Yor Processes. CVPR 2012: S. Ghosh & E. Sudderth, Nonparametric Learning for Layered Segmentation of Natural Images.

Page 2: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Human Image Segmentation

Page 3: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

BNP Image Segmentation

•! How many regions does this image contain? •! What are the sizes of these regions?

Segmentation as Partitioning

•! Huge variability in segmentations across images •! Want multiple interpretations, ranked by probability

Why Bayesian Nonparametrics?

Page 4: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

BNP Image Segmentation

Inference !!Stochastic search &

expectation propagation

Model !!Dependent Pitman-Yor processes

!!Spatial coupling via Gaussian processes

Results !!Multiple segmentations of

natural images

cesses

Learning !!Conditional covariance

calibration

Page 5: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Feature Extraction

•! Partition image into ~1,000 superpixels •! Compute texture and color features:

Texton Histograms (VQ 13-channel filter bank) Hue-Saturation-Value (HSV) Color Histograms

•! Around 100 bins for each histogram

Page 6: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Pitman-Yor Mixture Model

Observed features (color & texture)

Assign features to segments

PY segment size prior

Visual segment appearance model

Color: Texture:

π

z1 z2

z3z4

x1x2

x3x4xc

i ∼ Mult(θczi)

xsi ∼ Mult(θszi)

zi ∼ Mult(π)

πk = vk

k−1∏

�=1

(1− v�)

vk ∼ Beta(1− a, b+ ka)

Page 7: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Dependent DP&PY Mixtures

Observed features (color & texture)

Visual segment appearance model

Color: Texture:

z1 z2

z3z4

x1x2

x3x4xc

i ∼ Mult(θczi)

xsi ∼ Mult(θszi)

π1 π2

π3π4

Assign features to segments

zi ∼ Mult(πi)

Some dependent prior with DP/PY

“like” marginals

Kernel/logistic/probit stick-breaking process,

order-based DDP, !

Page 8: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Example: Logistic of Gaussians

•! Pass set of Gaussian processes through softmax to get probabilities of independent segment assignments

•! Nonparametric analogs have similar properties Figueiredo et. al., 2005, 2007

Fernandez & Green, 2002 Woolrich & Behrens, 2006 Blei & Lafferty, 2006

Page 9: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Discrete Markov Random Fields Ising and Potts Models

•! Interactive foreground segmentation •! Supervised training for known categories

Previous Applications

!but learning is challenging, and little success at unsupervised segmentation.

GrabCut: Rother, Kolmogorov, & Blake 2004

Verbeek & Triggs, 2007

Page 10: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Region Classification with Markov Field Aspect Models

Local: 74%

MRF: 78%

Verbeek & Triggs, CVPR 2007

Page 11: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

10-State Potts Samples

States sorted by size: largest in blue, smallest in red

Page 12: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

number of edges on which states take same value

1996 IEEE DSP Workshop

edge strength

Even within the phase transition region, samples lack the size distribution and spatial coherence of

real image segments

natural images

giant cluster

very noisy

Page 13: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Geman & Geman, 1984

200 Iterations

128 x128 grid 8 nearest neighbor edges K = 5 states Potts potentials:

10,000 Iterations

Page 14: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Product of Potts and DP? Orbanz & Buhmann 2006

Potts Potentials DP Bias:

Page 15: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Spatially Dependent Pitman-Yor Spatially D •! Cut random surfaces

(samples from a GP) with thresholds (as in Level Set Methods)

•! Assign each pixel to the first surface which exceeds threshold (as in Layered Models)

Duan, Guindani, & Gelfand, Generalized Spatial DP, 2007

π

z1 z2

z3z4

x1x2

x3x4

Page 16: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Spatially Dependent Pitman-Yor Spatially D Pitman-Yor •! Cut random surfaces

(samples from a GP) with thresholds (as in Level Set Methods)

•! Assign each pixel to the first surface which exceeds threshold (as in Layered Models)

Duan, Guindani, & Gelfand, Generalized Spatial DP, 2007

Page 17: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Spatially Dependent Pitman-Yor Spatially D Pitman-Yor •! Cut random surfaces

(samples from a GP) with thresholds (as in Level Set Methods)

•! Assign each pixel to the first surface which exceeds threshold (as in Layered Models)

•! Retains Pitman-Yor marginals while jointly modeling rich spatial dependencies (as in Copula Models)

Page 18: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Stick-Breaking Revisited

0 1

Multinomial Sampler: Sequential Binary Sampler:

Page 19: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

PY Gaussian Thresholds

Sequential Binary Sampler: Gaussian Sampler:

Normal CDF

because

Page 20: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

PY Gaussian Thresholds

Sequential Binary Sampler: Gaussian Sampler:

Page 21: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Spatially Dependent Pitman-Yor Spatially D Non-Markov Gaussian Processes:

PY prior: Segment size

Feature Assignments

Normal CDF

Page 22: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Preservation of PY Marginals Preserva Why Ordered Layer Assignments?

ation of Why Ordered L

Stick Size Prior Random Thresholds

Page 23: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Samples from PY Spatial Prior

Comparison: Potts Markov Random Field

Page 24: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Outline

Inference !!Stochastic search &

expectation propagation

Model !!Dependent Pitman-Yor processes

!!Spatial coupling via Gaussian processes

Results !!Multiple segmentations of

natural images

cesses

Learning !!Conditional covariance

calibration

Page 25: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Mean Field for Dependent PY

K

K

Factorized Gaussian Posteriors

Sufficient Statistics

Allows closed form update of via

Page 26: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Mean Field for Dependent PY

K

K

Updating Layered Partitions Evaluation of beta normalization constants:

Jointly optimize each layer’s threshold and Gaussian assignment surface, fixing

all other layers, via backtracking conjugate gradient with line search

Reducing Local Optima Place factorized posterior on eigenfunctions

of Gaussian process, not single features

Page 27: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Robustness and Initialization

Log-likelihood bounds versus iteration, for many random initializations of mean field variational inference on a single image.

Page 28: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Alternative: Inference by Search

Consider hard assignments of

superpixels to layers (partitions) Integrate

likelihood parameters analytically (conjugacy)

Marginalize layer support functions via expectation propagation (EP): approximate but very accurate

No need for a finite, conservative model truncation!

Page 29: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Maximization Expectation EM Algorithm !!E-step: Marginalize latent variables (approximate)

! M-step: Maximize likelihood bound given model parameters

ME Algorithm !!M-step: Maximize likelihood given latent assignments

! E-step: Marginalize random parameters (exact)

Kurihara & Welling, 2009

Why Maximization-Expectation? !!Parameter marginalization allows Bayesian “model selection”

!!Hard assignments allow efficient algorithms, data structures

!!Hard assignments consistent with clustering objectives

!!No need for finite truncation of nonparametric models

Page 30: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Discrete Search Moves

!!Merge: Combine a pair of regions into a single region

!!Split: Break a single region into a pair of regions (for diversity, a few proposals)

!!Shift: Sequentially move single superpixels to the most probable region

!!Permute: Swap the position of two layers in the order

Stochastic proposals, accepted if and only if they improve our EP estimate of marginal likelihood:

Marginalization of continuous variables simplifies these moves!

Page 31: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Inferring Ordered Layers

Order A: Front, Middle, Back Order B: Front, Middle, Back

!!Which is preferred by a diagonal covariance?

!!Which is preferred by a spatial covariance?

Order B

Order A

Page 32: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Inference Across Initializations

Mean Field Variational EP Stochastic Search

Best Worst Best Worst

Page 33: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

BSDS: Spatial PY Inference Sp

atia

l PY

(EP)

Sp

atia

l PY

(MF)

Page 34: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Outline

Inference !!Stochastic search &

expectation propagation

Model !!Dependent Pitman-Yor processes

!!Spatial coupling via Gaussian processes

Results !!Multiple segmentations of

natural images

cesses

Learning !!Conditional covariance

calibration

Page 35: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Covariance Kernels •! Thresholds determine segment size: Pitman-Yor •! Covariance determines segment shape:

Roughly Independent Image Cues:

Berkeley Pb (probability of boundary) detector

probability that features at locations are in the same segment

!!Color and texture histograms within each region: Model generatively via multinomial likelihood (Dirichlet prior)

! Pixel locations and intervening contour cues: Model conditionally via GP covariance function

Page 36: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Learning from Human Segments

!!Data unavailable to learn models of all the categories we’re interested in: We want to discover new categories!

! Use logistic regression, and basis expansion of image cues, to learn binary “are we in the same segment” predictors:

!! Generative: Distance only

!! Conditional: Distance, intervening contours, !

Page 37: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

From Probability to Correlation

There is an injective mapping between covariance and the probability that two superpixels are in

the same segment.

Page 38: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Low-Rank Covariance Projection

!! The pseudo-covariance constructed by considering each superpixel pair independently may not be positive definite

!!Projected gradient method finds low rank (factor analysis), unit diagonal covariance close to target estimates

Page 39: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Prediction of Test Partitions

Heuristic versus Learned Image Partition Probabilities

Learned Probability versus Rand index measure of partition overlap

Page 40: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Comparing Spatial PY Models

Image PY Learned PY Heuristic

Page 41: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Outline

Inference !!Stochastic search &

expectation propagation

Model !!Dependent Pitman-Yor processes

!!Spatial coupling via Gaussian processes

Results !!Multiple segmentations of

natural images

cesses

Learning !!Conditional covariance

calibration

Page 42: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Other Segmentation Methods

FH Graph Mean Shift NCuts gPb+UCM Spatial PY

Page 43: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Quantitative Comparisons

Berkeley Segmentation LabelMe Scenes !!On BSDS, similar or better than all methods except gPb

! On LabelMe, performance of Spatial PY is better than gPb

!! Implementation efficiency and search run-time

!!Histogram likelihoods discard too much information

!!Most probable segmentation does not minimize Bayes risk

Room for Improvement:

Page 44: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Multiple Spatial PY Modes

Most Probable

Page 45: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Multiple Spatial PY Modes

Most Probable

Page 46: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Spatial PY Segmentations

Page 47: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Conclusions !! efficient variational parsing of scenes

into unknown numbers of segments

!! empirically justified power law priors

!! accurate learning of non-local spatial statistics of natural scenes

!! promise in other application domains!

Spatial Pitman-Yor Processes allow!

Page 48: Applied Bayesian Nonparametrics 5. Spatial Models via ...sudderth/bnpCVPR12/slides/BNP5spatial.pdfApplied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs

Conclusions !!Conventional MCMC & variational

learning prone to local optima, hard to scale to large datasets. But better methods on the way!

!! Literature remains fairly technical. But growing number of tutorials!

!but bravery is required