Image Matting and Its Applications
description
Transcript of Image Matting and Its Applications
![Page 1: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/1.jpg)
Image Matting and Its Applications
Chen-Yu TsengAdvisor: Sheng-Jyh Wang
2012-10-29
![Page 2: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/2.jpg)
Image Matting
• A process to extract foreground objects from an image, along with an alpha matte (the opacity of the foreground color)
Input Image Alpha Matte Extracted Foreground
![Page 3: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/3.jpg)
Two Approaches of Image Matting
• Supervised Matting• With User’s Guidance
• Unsupervised Matting• Without User’s Guidance
Input Image User’s Guidance
e.g. Trimap:White ForegroundBlack BackgroundUnknown Gray
![Page 4: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/4.jpg)
Two Schemes of Supervised Matting
Propagation-based Scheme• Infer Alpha Matte with
Propagation through a Graphical Model
• A Global-based Approach
Sampling-based Scheme• Infer Alpha Matte with
Some Color Samples• A Local-based Approach
Foreground Pixel
Background Pixel
Unknown Pixel
Foreground Color Set
Background Color Set
Unknown Pixel
![Page 5: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/5.jpg)
Propagation-based scheme -Matting Laplacian Approach
• A Graphical Model with Connectivity between Pixels• The Connectivity Is Learned from the Image Structure
• Capability for Dealing with Both • Supervised Matting (Inference Problem)• Unsupervised Matting (Decomposition Problem)
Foreground Pixel
Background Pixel
Unknown Pixel
![Page 6: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/6.jpg)
Reference of Matting Laplacian Approach
• First proposed by Levin et al. for supervised matting (closed-form matting)• A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural
Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008.• Extended to unsupervised matting (spectral matting)
• A. Levin, A. Rav-Acha, D. Lischinski. “Spectral Matting,” IEEE T. PAMI, vol. 30, no. 10, pp. 1699-1712, Oct. 2008.
• Extended to learning-based matting• Y. Zheng and C. Kambhamettu. “Learning based digital matting,” In
ICCV, pages 889–896, 2009.• Extended to multi-layer matting
• D. Singaraju, R. Vidal. “Estimation of Alpha Mattes for Multiple Image Layers,” IEEE T. PAMI, vol. 33, no. 7, pp. 1295-1309, July 2011.
![Page 7: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/7.jpg)
Matting Laplacian
Input Image
EstimatingPair-wise Affinity
Graphical ModelNode: Image PixelsEdge: Affinity
Supervised Matting
Background
Foreground
Matting Laplacian Matrix:Recording the Connectivity between Pair of Pixels
![Page 8: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/8.jpg)
Introduction of Graph Laplacian
2
3
1
4
5
A Graph with Five Vertexes
: Adjacency Matrix
𝐿=𝐷−𝑊: Laplacian Matrix
𝑊=(𝑤𝑖𝑗)𝑖 , 𝑗=1 ,… ,𝑛
: Degree Matrix
𝑑𝑖𝑖=∑𝑗=1
𝑛
𝑤𝑖𝑗
0 1 1 0 01 0 1 0 01 1 0 0 00 0 0 0 10 0 0 1 0
12345
1 2 3 4 5
: Adjacency Matrix
Vertex Index
![Page 9: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/9.jpg)
Cutting Cost Function with Graph Laplacian
𝜶𝑇 𝐿𝜶=12 ∑𝑖 , 𝑗=1
𝑛
𝑤𝑖𝑗 (𝛼 𝑖−𝛼 𝑗 )2
Cost Function for Cutting Criterion
Low-costAssignment
High-costAssignment2
3
1
4
5
2
3
1
4
5
![Page 10: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/10.jpg)
Construction of Matting Laplacian
• Color-model-based Approach (Original)• Estimating Affinity Based on Relative Color Distance
• Learning-based Approach (Extended)• Learning Affinity Based on Image Structure
![Page 11: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/11.jpg)
Construction of Matting LaplacianColor-model-based Approach
Color Distribution
𝐼 𝑖
𝐼 𝑗
𝜇𝑘
Input Image
A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008.
gr
b
![Page 12: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/12.jpg)
Construction of Matting LaplacianLearning-based Approach
• Learning Affinity among Local Pixels
¿ [𝐱 𝑖𝑇 1 ] [ 𝜷𝛽0 ]
Linear Alpha-color Model for Single Pixel:
: Alpha Value for Pixel i: Feature Vector (): Linear Coefficient
Extending to a Local Patch qAssuming all Pixels Sharing the Same Linear Coefficient 𝛼 𝑖=𝐱 𝑖
𝑇 𝜷+𝛽0�⃗�𝑞=𝐗𝒒
𝑇 [ 𝜷𝛽0]: Alpha Vector for Patch q: Feature Matrix: Linear Coefficient
![Page 13: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/13.jpg)
Construction of Matting LaplacianLearning-based Approach
[ 𝜷𝛽0]=argmin𝜷 ,𝛽0‖�⃗�𝑞−𝐗𝒒𝑇 [ 𝜷𝛽0]‖
2
+𝜆𝑟 𝜷𝑇 𝜷
¿ (𝐗𝒒𝑇𝐗𝒒+𝜆𝑟 𝐈 )−𝟏𝐗𝒒 �⃗�𝑞
�⃗�𝑞=𝐗𝒒𝑇 [ 𝜷𝛽0]
¿𝐗𝒒𝑇 (𝐗𝒒
𝑇𝐗𝒒+𝜆𝑟 𝐈 )−𝟏𝐗𝒒 �⃗�𝑞
Derived Linear Coefficient
Rewritten Linear Model
![Page 14: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/14.jpg)
Construction of Matting LaplacianLocal Cost Function
�⃗�𝑞=𝐗𝒒𝑇 (𝐗𝒒
𝑇𝐗𝒒+𝜆𝑟 𝐈 )−𝟏𝐗𝒒 �⃗�𝑞
Local Cost Function
¿ �⃗�𝑞𝑇 𝑳𝑞 �⃗�𝑞
: Local Laplacian Matrix for Patch qInput Image
Patch q
Local Linear Model
![Page 15: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/15.jpg)
Construction of Matting LaplacianLocal Global
Local Cost Function
¿ �⃗�𝑞𝑇 𝑳𝑞 �⃗�𝑞
: Local Laplacian Matrix for Patch q
Input Image
Patch q
Global Cost Function
¿ �⃗�𝑇 𝑳 �⃗�
![Page 16: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/16.jpg)
Supervised Matting (Closed-form Matting)
Foreground Pixel
Background Pixel
Unknown Pixel
Input Image User’s Guidance,
𝐸 ( �⃗� )=�⃗�𝑇 𝑳 �⃗�+(�⃗�− �⃗�)𝑇𝚲 (�⃗�− �⃗�)
Foreground
Background
Unknown
1 0 -1 1 0
Cost Function for Supervised Matting
Affinity Cost Data Cost
�⃗�∗=(𝑳+𝚲 )−1𝚲 �⃗�Optimal Solution
![Page 17: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/17.jpg)
Experimental Results
Input Image Alpha Matte Synthesized Result
![Page 18: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/18.jpg)
Unsupervised Matting (Spectral Matting)
• Solving Alpha Matte without User’s Guidance• Procedures
• Decomposing Image into Several Matting Components• Combining Matting Components into Alpha Matte
![Page 19: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/19.jpg)
Spectral Clustering
s.t. =1 𝐿 𝒇 =λ 𝒇1. L is symmetric and positive semi-definite.2. The smallest eigenvalue of L is 0, the
corresponding eigenvector is the constant one vector 1.
3. L has n non-negative, real-valued eigenvalues
0= λ 1 ≦ λ 2 ≦ . . . ≦ λ n.
: Eigenvector: Eigenvalue
2
3
1
4
5
A Graph Example
2 -1 -1 0 0-1 2 -1 0 0-1 -1 2 0 00 0 0 1 -10 0 0 -1 1
12345
1 2 3 4 5
: Laplacian Matrix
0.0470.0470.0470.0470.047
0.5770.5770.57700𝒇 1 𝒇 2
![Page 20: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/20.jpg)
Spectral Clustering & Matting Components
2 -1
-1
0 0 0 0
-1
2 -1
0 0 0 0
-1
-1
2 0 0 0 0
0 0 0 1 1 0 00 0 0 -
1-1
0 0
0 0 0 0 0 1 10 0 0 0 0 -
1-1
: Laplacian Matrix
1110000
0001100
0000011
Zero-Eigenvectors Binary Indicating Vectors
×𝑹3×3Linear
Transformation
![Page 21: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/21.jpg)
Overview of Spectral Matting
Input Image
Smallest Eigenvectors
Matting Components
K-means Clustering
&Linear
TransformationMatting
Laplacian
![Page 22: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/22.jpg)
Spectral Clustering & K-means
Input Image
s-smallest Eigenvectors
…
Pixel i
s-dimensional
Space
K-means Clustering
![Page 23: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/23.jpg)
Generating Matting Components
Smallest Eigenvectors
Projection into Eigen Space
..K-means .… … …
𝑬=[𝒆𝟏 … 𝒆𝒔 ] 𝒎𝒌 𝜶𝒌=𝑬 𝑬𝑻𝒎𝒌
![Page 24: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/24.jpg)
Reconstructing Alpha Matte from Matting Components
=+ +
Input Image
Matting Components
Selected Matting Components
Alpha Matte
![Page 25: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/25.jpg)
Reconstructing Alpha Matte by Grouping Matting Components
Matting cost function
𝐽 ( �⃗� )=�⃗�𝑇 𝑳 �⃗�
�⃗�=[ �⃗�1 … �⃗�𝑘 ] �⃗�Alpha Matte Generation
: Combination Vector
¿ �⃗�𝑇 [ �⃗�1 … �⃗�𝑘 ]𝑇 𝑳 [�⃗�1 … �⃗�𝑘 ] �⃗�¿ �⃗�𝑇𝜱�⃗�
Evaluating All Grouping Hypothesis to Derive the Optimal Alpha Matte
![Page 26: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/26.jpg)
Results by Levin et al.
![Page 27: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/27.jpg)
Summary
• Constructing Matting Laplacian• Solving Supervised Matting Problem• Solving Unsupervised Matting Problem
![Page 28: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/28.jpg)
Proposed Approaches
• Efficient Cell-based Framework for Reducing Computations• Multi-scale Analysis• Extended Applications (Depth Image Reconstruction)
Input Image Reconstructed Depth
Depth Reconstruction from Single Image
Depth Reconstruction in Shape From Focus (SFF)
Input Image Reconstructed Depth
![Page 29: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/29.jpg)
Cell-based Framework
Image
Pixel-wise Data Distribution
Cell-wise Data Distribution
ConventionalMatting Laplacian
Cell-basedMatting Laplacian
Pixel-wise Affinity
Cell-wise Affinity
![Page 30: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/30.jpg)
Multi-scale Affinity Learning
Image & Computation Patterns
Pixel-based Approach
Cell-based Approach
![Page 31: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/31.jpg)
Multi-scale Affinity Learning
…Finest Level Coarsest
Level …
Cell-based Graph
![Page 32: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/32.jpg)
Results of Reconstructed Alpha Matte
1st Rank 2nd Rank
(a) Grouping Results by Levin et al.
(b) Grouping Results by Levin et al. with Coarse-to-fine Scheme.
(c) Ours
Input
![Page 33: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/33.jpg)
Results
(a) Input images
(b) Levin’s result (c) Our result
![Page 34: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/34.jpg)
Proposed Approaches
• Efficient Cell-based Framework for Reducing Computations• Multi-scale Analysis• Extended Applications (Depth Image Reconstruction)
Input Image Reconstructed Depth
Depth Reconstruction from Single Image
Depth Reconstruction in Shape From Focus (SFF)
Input Image Reconstructed Depth
![Page 35: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/35.jpg)
Depth Reconstruction in Shape From Focus (SFF)
Optical Direction
Multi-focus Image Sequence
Optical Direction
FocusValueW1
W2
W2
W1
![Page 36: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/36.jpg)
Low-SNR Problem
• Spatially Varying Precision• Low-texture Low-SNR• Leading Noisy Result
Input Image Observation
High-precision
Low-precision
![Page 37: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/37.jpg)
Proposed Maximum-a-posteriori Estimation
Multi-focus Image Sequence
Learning-based Graph
Local Learning
Inference
Reconstructed Depth
![Page 38: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/38.jpg)
Proposed Maximum-a-posteriori Estimation
𝐷∗=max (𝑝 (𝐷|𝑌 , 𝐼 ) ): Optimal Result: Depth Image: Observation: Input Image
Posterior Likelihood Prior
Local Observation with Spatial-varying Precision
Learned from Image
![Page 39: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/39.jpg)
Likelihood Model
Input Observation
Precision Result
High-precision
Low-precision
𝑰 𝒀
𝚲 𝐷∗
Posterior Likelihood Prior
Local Observation with Spatial-varying Precision
![Page 40: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/40.jpg)
Prior Model Posterior Likelihood Prior
Learning from Input Image
Learning-based Graph
Local Learning
Multi-focus Image Sequence
![Page 41: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/41.jpg)
Maximum-a-posteriori Estimation for Depth Reconstruction
𝐷∗=max (𝑝 (𝐷|𝑌 , 𝐼 ) )
− log𝑝 (𝐷|𝑌 , 𝐼 )∝ ( �⃗�−𝒚 )𝑇𝚲 ( �⃗�− �⃗� )+ �⃗�𝑇 𝑳 �⃗�
𝐷∗= (𝑳+𝚲 )−1𝚲 �⃗�
Input Image Observation Reconstructed Depth
![Page 42: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/42.jpg)
Results of Shape from Focus
Input Image M. Mahmood, 2012 T. Aydin, 2008 OursS. Nayar, 1994
![Page 43: Image Matting and Its Applications](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681642f550346895dd5fa3a/html5/thumbnails/43.jpg)
Conclusions
• Construction of Matting Laplacian• Conventional Approach• Multi-scale Cell-based Approach
• Supervised Matting• Spectral Matting• Depth Reconstruction